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(57) ABSTRACT 

A color image input from a document image inputting unit 
is converted into a gray-scale image by a brightness image 
extracting unit. The gray-scale image is then converted into 
an image having a higher resolution according to the reso- 
lution of the original gray-scale image. When this conver- 
sion is performed, subpixels are generated between the 
original pixels, and the values of the subpixels are obtained 
with an interpolation method. Furthermore, a threshold 
value for a binarization process is generated by using an 
original pixel value and a subpixel value. The characters 
included in the binarized image are recognized by a binary 
image recognizing unit, and a recognition result is output 
from a recognition result outputting unit. 
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DEVICE, METHOD AND STORAGE data having the same brightness data as that of the input 

MEDIUM FOR RECOGNIZING A gray-scale image, in principle. This is because only the 

DOCUMENT IMAGE parameters related to the hue are removed from all the color 

parameters of the processed gray-scale image. 

BACKGROUND OF THE INVENTION 5 if a gray-scale image which is a brightness image is input, 

1 Field of the Invention a P re d etermme d threshold value binarizing unit 172 obtains 

' t . 1 * * • - • a binary image by binarizing the gray-scale image with a 

Tie present invent.*, relates to an .mage recognizing pnaa ^ a ^ ^^16 value. TO. threshold value is a 

evicc * value which is externally determined and input. Hereinafter, 
2. Description of the Related Art 10 a gray-scale image fundamentally indicates not the gray- 

With the popularization of personal computers and the scale image with color parameters set to black-and-white, 

arrangements of networks, the number of electronic docu- which is scanned by the document image inputting unit 170, 

ments has been growing in recent years. However, the main but the brightness image resultant from the process per- 

medium of information distribution is still a paper formed by the brightness image extracting unit 171. That is, 
document, and an enormous number of paper documents *5 the gray-scale image is defined to have not color but 

currently exist. Accordingly, a document image recognizing brightness parameters. Even if the gray-scale image has 

device intended to convert a paper document into an elec- color parameters, only brightness parameters are substan- 

tronic document, and to edit a conversion result has been tially valid for recognizing the image if the color parameters 

increasingly demanded. The document image recognizing are set to black-and-white for all of the pixels. Accordingly, 
device is a device which uses a document image as an input, 20 image recognition can be made also by using such a gray- 

and performs a coding process by recognizing characters, scale image. 

etc. included in the document image. Currently, there are A binary image recognizing unit 173 recognizes a binary 

products which use a binary document image as an input. image. That is, this unit reco gnizes characters by obtaining 

The binary document image is a document image where the features of the document image which is Binarized by the 
image data of a character is represented, for example, by 25 predetermined threshold value binarizing unit 172, and 

"1", while the image data of the background is represented replaces the characters with the codes which are internally 

by "0"- used by a computer and correspond to the recognized i 

Recently, an image filing device has become popular, and characters. ' — 
also the demand for inputting an image filed by the image A recognition result outputting unit 174 outputs the result 

filing device to a document image recognizing device, and 0 f trie character recognition made by the binary image 

for recognizing the image has been increasing. Especially, recognizing unit 173, that is, the file which is restructured 

the number of gray-scale or color documents including mt0 a C(X j e sequence representing the characters of a docu- 

photographs has been growing. Therefore, the demand for menl image 

recognizing not only binary documents but also gray-scale The ^y^^ color documen t image recognizing 

or color documents has been on the rise. device has a disadvantage that recognition accuracy is low. 

A color document image recognizing device recognizes ^ ^ rf dm rf 6ach ^ rf a ^ docu . 

also a color or gray -scale document image. A conventional . . • o ** *u * c u* +u i 

" J ...... i_- ment image is 8 times that ot a binary image, the color 

color document .mage recognmng device obtains a binary document ^ is { tQ a ^ document ^ 

.mage by binarmng each bnghtoess component with a w ^ ^vice at a resolution Lower than that of the binary 

predetermined threshold value, and recognizes the obtained ^ ordef tQ reduce an amount of processmg time of ^ 

bmary image, if an input document .mage is not a binary ^ ^ devioe ^ „ a x etc and a cit of 

image but a grayscale or color .mage. a memory used for ^ m ^ 

FIGS. 1A and IB respectively show the configuration of + n ... , A , , , . 4 . i * * 

t . . * , K \. • » . FIG. IB illustrates an expanded low-resolution color text 

the conventional color document image recognizing device 4S . r 

and an extended color text image. image. 

i nr- ~t a j „ . • ... -* ■ •* This figure shows a monochromatic image of a 150-dpi 

In FIG. 1A, a document image inputting unit 170 is a unit •*„,«»■. , * 

, . . , t . ... . tl . text image in full color. With a color display, many colors 

for inputting a document image, and is typically imple- , * , . . « . , u 

monta A nc n ™„„ Af . ^ ^TLw j^^,, mo „. can be identified on the periphery of the characters, and it 

me nted as a scanner, etc. b or a color document, parameters ~ t X , it _ 

, . ■ • « / seems mfncult to identify an area by extracting the same 

such as a color parameter, a brightness parameter, etc. are 50 . 7 7 6 

assigned to respective pixels by illuminating the document, 00 or * 

receiving a reflected light, and analyzing the received light. The conventional document image recognizing device is 

If the document to be scanned is a gray-scale document, the fundamentally designed to have an input of a binary image 

light reflected from the document is analyzed, the informa- ^th a small amount of data, and assumes the document 

lion about the level of brightness is obtained, and this 5S "^S 0 of a standard resolution of approximately 400 dpi. 

information is assigned to each pixel. At this time, all of the Accordingly, if a color document image of 150 or less dpi is 

pixels of the gray-scale document are detected to be black- m P ut ' the conventional device converts the document image 

and-white, which is set as the color parameters of each of the a binarv ™ ih the resolution equal to or less than 

pixels 150 dpi, and recognizes the converted image. Therefore, the 

if : , ■ ^ • n • Q u«„kf«™ ; ^ cn device cannot recognize the image with sufficient accuracy. 

It an input image is a color image, a brightness image 60 & & j 

extracting unit 171 extracts a brightness component for each SUMMARY OF THE INVENTION 
pixel, and outputs a brightness image which is a gray-scale 

image to a predetermined threshold value binarizing unit. If An object of the present invention is to provide a high- 

the input image is a gray-scale image, the color parameters speed document image recognizing device which imple- 
of all of the pixels are set to be black-and-white. Therefore, 65 ments high recognition accuracy. 

the gray-scale image resultant from the process of the An image recognizing device according to the present 

brightness image extracting unit 171 will become the image invention comprises: an image converting unit for convert - 
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ing an input document image into a gray-scale image if the FIG. 15 exemplifies a sixth configuration of the locally 

input document image is a color image, and for newly binarizing unit; 

outputting a gray-scale image if the input document image p]G. 16 exemplifies a seventh configuration of the locally 

is a gray-scale image; a variable resolution binarizing unit binarizing unit; 

for converting the gray-scale image into a binary image with 5 nG 1? fa a ^ chematic diagram exemplifying the process- 

a higher resolution according to the resolution of the gray- • to thc ^ for 5inarizing a ^ or g^y-scale 

scale image; and a unit for recognizing the binarizcd image. image ^ tfac proccss according t0 the prcfcrred embodi . 

An image recognizing method according to the present ment (No. 1); 
invention comprises the steps of: (a) converting an input F , G lg ^ [ di exemplifying the process- 
document image into a gray-scale image if the input docu- 10 tQ ^ ^ fof binarizin a color Qf 
ment image is a color image, and for newly outputting a ^ ^ ^ ^ accordm to the ferred em5odi . 
gray-scale image if the input document image is a gray-scale mei j 2 \. ^ 

image; (b) converting the gray-scale image into a binary !■ ' } * . . . 

image with a higher resolution according to the resolution of u FI ?* * 9 15 a ^ock diagram explaining the configuration of 

the gray-scale image; and (c) recognizing the binarized « ^e hardware retired for implementing the preferred 

• _ J 6 w & & embodiment as software, 
image. 

According to the present invention, a gray-scale image DESCRIPTION OF THE PREFERRED 

obtained by converting a color document image or an input EMBODIMENTS 
gray-scale image is converted into the image data having a 

suitably higher resolution according to the resolution of the 20 According to the present invention, an input image is 

gray-scale image, thereby binarizing the image without converted into a binary image with a higher resolution 

losing the information about brightness levels of a gray according to the resolution of the input image m order to 

scale. Therefore, characters appearing in a binary image can overcome the above described problems of the conventional 

be prevented from being defaced, thereby implementing device. 

character recognition with higher accuracy. 25 Even if the resolution of an input image is low, a human 

Furthermore, a drawing area including characters, etc. is be ing can recognize the image if it is a color or gray-scale 

roughly extracted from a document image, and a binariza- image. However, even a human being has difficulty in 

tion process according to the present invention is performed recognizing an image which is binarized by the conventional 

only for the extracted area, thereby improving the processing device. This is because the information about the brightness 

speed. 30 components of the original image is lost. To facilitate the 

recognition of the binarized image, the information about 

BRIEF DESCRIPTION OF THE DRAWINGS me brightness components of the image must be reflected in 

FIG. 1A shows the configuration of a conventional color the binarized image. For the implementation of the 

document image recognizing device; reflection, it is effective to increase the resolution of the 

FIG. IB shows an expanded color text image; 35 binarized image. 

FIG. 2 is a block diagram showing the principle of a color If an ^ document image is a color image, it is 

document image recognizing device according to a preferred converted into a gray-scale image and further converted into 

embodiment* a binary image with the resolution according to that of the 

FIGS. 3A and 3B exemplify the configurations of a 40 ^i^^^^^^^^^r^f^^ 

. «. * • • • • • 0 the characters are converted into electronic codes, 

variable resolution binanzing unit; . . 

™~ A . . j. _ i--.u ■ • 1 * As a specific means for converting a gray -scale image into 

FIG. 4 is a schematic diagram explaining toe principle of t . \ .... , *• , 7 r 

. r . 0 ,_. , 0 a binary image having the resolution according to that 01 an 

the process for generating subpixels; . ' & iL & , - . . • f * 

Jl„ m * . / ... ._ * m mput image, a method for obtaining a binary image after 

FIG. 5 exemphfies the able for specifying the number and formi thc ^ ixcl gcneratioD process which increases 

the resolution of subpixels to be generated according to the 45 ^ Qf ^ a ale { b 

resolution of an input gray-scale image; interpolating the values of the pixels included in the gray- 

FIG. 6 exemplifies an additional configuration of the mU -^ge, is considered. As a specific method of the 

variable resolution binanzing unit; subpixel generation process, a linear interpolation method 

FIG. 7 is a schematic diagram explaining the method for between pixel values can be cited, 

setting a global threshold value used for a rough extraction 50 the method for extracting a character por- 

process; ^ on and per iphery from an entire image, for suitably 

FIGS. 8A and 8B exemplify the further configurations of generating a threshold value within the extracted partial 

the variable resolution binarizing unit; image, and for performing character recognition, can be 

FIG. 9 exemplifies the details of a first configuration of a 55 c jted as a method for improving the recognition ratio of a 

locally binarizing unit; color or gray-scale image. With this method, the result of the 

FIG. 10 exemplifies the details of a second configuration binarization process allows a character to be clearly shaped 

of the locally binarizing unit; by reducing the light and shade of an area having a higher 

FIG. 11 exemplifies a third configuration of the locally level of brightness in the background, that is, noise compo- 

binarizing unit; 60 nents rather than by preventing the information of brightness 

FIG. 12 exemplifies a fourth configuration of the locally components from being lost, 

binarizing unit; FIG. 2 is a block diagram showing the principle of a color 

FIG. 13 exemplifies a fifth configuration of the locally document image recognizing device according to a preferred 

binarizing unit; embodiment. 

FIG. 14 is a schematic diagram explaining the method for 65 A document image inputting unit 10 is a unit correspond- 

calculating a local average value and a local square average ing to a scanner in a similar manner as in the above described 

value from original pixel values; conventional technique, and is intended to illuminate a color 
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or gray-scale document (including parameters related to 
hue), and to capture the document as an image. 

A brightness image extracting unit 11 is a unit for extract- 
ing only brightness components from the image input by the 
document image inputting unit 10, and for generating a 
gray-scale image (which does not include parameters related 
to hue). The gray-scale image which is the output of the 
brightness image extracting unit 11 is input to a variable 
resolution binarizing unit 12 to be described next. 

The variable resolution binarizing unit 12 (roughly) 
extracts a partial area including characters from the input 
gray-scale image having a low resolution, and provides a 
binary image where characters are easily recognized to a 
binary image recognizing unit 13 at a succeeding stage by 
generating subpixels between original pixels within the 
gray-scale image and interpolating the information about 
brightness components. 

The binary image recognizing unit 13 performs character 
recognition based on the binary image passed from the 
variable resolution binarizing unit 12, and performs the 
process for replacing characters of an image with electronic 
codes. 

A recognition result outputting unit 14 receives the docu- 
ment file as an electronic code sequence from the binary 
image recognizing unit 13, stores the document file onto a 
storage medium such as a hard disk, etc., and outputs the 
document file as a recognition result to a display monitor 

Since the configurations of the document image inputting 
unit 10, the brightness image extracting unit 11, the binary 
image recognizing unit 13, and the recognition result out- 
putting unit 14 are similar to those of the conventional 
technique, their detailed explanations are omitted here. That 
is, even if they are omitted, a skilled artisan can easily 
understand the configurations of the document image input- 
ting unit 10, the binary image recognizing unit 13, and the 
recognition result outputting unit 14, and can actually use 
them in a current situation where devices and software for 
recognizing a black-and-white document are commercial- 
ized. Additionally, the skilled artisan can easily understand 
also the configuration of the brightness image extracting unit 
11, because this unit is intended to convert a color image, 
etc. into a gray-scale image, and, at present, a color image 
is converted into a black-and-white image and telecast. 
Therefore, its explanation is omitted here. 

Accordingly, the present invention is characterized in that 
a predetermined threshold value binarizing unit 172 is 
replaced with a variable resolution binarizing unit 12. The 
explanation to be provided below will refer to the details of 
the variable resolution binarizing unit 12. Hie variable 
resolution binarizing unit 12 converts a gray-scale image 
into the binary image having the resolution according to that 
of an input image, or binarizes a gray-scale image after 
roughly extracting a portion including characters from a 
gray-scale image, so that the image can be used for character 
recognition. Additionally, the resolution conversion and the 
rough extraction may be performed at the same time. 

FIGS. 3A and 3B are block diagrams exemplifying the 
configurations of the variable resolution binarizing unit. 

As a first example of the configuration of the variable 
resolution binarizing unit, subpixels are generated between 
original pixels of a gray -scale image by a subpixel process- 
ing unit 20, the resolution is increased according to the 
resolution of the gray-scale image which is an input image, 
and then the conventional process for binarizing an image 
with a predetermined threshold value is performed by a 
predetermined threshold value binarizing unit 21, as shown 
in FIG. 3A. 
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Here, a predetermined threshold value is defined to be 
used for a single document image. For example, only one 
threshold value is used for binarizing a 1-page document 
image. 

5 The process for generating subpixels is a process for 
subdividing the space between original pixels of a gray-scale 
image whose recognition ratio is not likely to be improved 
if it is binarized unchanged, and for generating virtual pixel 
data in the original data. Although its details will be 

10 described later, in short, the pixel value (the level of 
brightness) of a subpixel is obtained by interpolating the 
levels of brightness of original pixels of an input gray-scale 
image. Typical of the interpolation method is a linear inter- 
polation method. 

15 As a second example of the configuration of the variable 
resolution binarizing unit, a locally binarizing unit 23 shown 
in FIG. 3B may be arranged instead of the predetermined 
threshold value binarizing unit 21 shown in FIG. 3A, as 
illustrated in FIG. 3B. The locally binarizing unit 23 sets the 

20 local area where the process is to be performed for each 
pixel included in a drawing area within an entire gray-scale 
document image, and obtains a binary image by binarizing 
the image with the threshold value generated by using the 
pixel data within the local area. 

25 

That is, the threshold value of the level of brightness is 
obtained in a local range (local area such as a regular square 
area centering around a target pixel), and the binarization 
process is performed. That is, the binarization process is 

30 performed so that the pixel with the level of brightness 
which is equal to or lower than the threshold value is black 
(the level of brightness is set to, for example, "1"), and the 
pixel with the level of brightness which is equal to or higher 
than the threshold value is white (the level of brightness is 

3S set to, for example, "0"). The threshold value of a local area 
is defined, for example, with the method using the linear 
combination of an average pixel value, a standard deviation 
value, and a variance. The threshold value is defined, for 
example, as follows. In the following equation, a local 

^ binarization parameter is a constant, and an optimum thresh- 
old value is obtained by suitably setting this parameter. Note 
that the gray scale is used almost similar to the brightness in 
the following equation. 

(threshold value)-(average gray scale)+(tocal binarization param- 
45 ctcr)x (gray-scale standard deviation value) 

FIG. 4 is a schematic diagram explaining the principle of 
the process for generating subpixels. 

In this figure, black circles indicate original pixels of a 

50 gray-scale image, while white circles indicate subpixels. 
Additionally, IA through ID indicate the levels of the gray 
scale of the original pixels A through D. Furthermore, II 
through 15 indicate the levels of gray scale of the subpixels 
1 through 5 to be obtained from the levels of gray scale of 

55 the original pixels with the interpolation process. 

As shown in this figure, if subpixels are generated with 
the linear interpolation within an area enclosed by the four 
original pixels, it is first determined how many subpixels 
will be arranged between the original pixels. Next, subpixels 

60 are positioned at regular intervals according to the number 
of subpixels to be arranged. Then, the levels of gray scale are 
assigned to the respective subpixels by interpolating the 
levels of gray scale of the original pixels. 
Considered below is the case where the subpixels are 

65 arranged in the area enclosed by the original pixels A 
through D based on the assumption that "p" and "q" are 
numbers which are larger than "0" and smaller than "1". A 
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subpixel 1 is arranged at the position represented by "p:l-p" perform the global process for recognizing as the periphery 

in a straight line AB unking the original pixels A and B. In of characters the image portion whose pixel values are lower 

this case, the levels of gray scale of the subpixel 1 is than a global threshold value set for the pixel values of a 

obtained from the levels of gray scales IA and IB of the gray-scale image, and for roughly extracting the drawing 

original pixels A and B with the linear interpolation accord- 5 area. Then, a local binarization process may be performed, 

ing to the following equation. That is, the drawing area roughly extracting unit 50 

extracts the portion including characters and its periphery 

iUp*/B+(i-pyiA fjQUj a document image, and the locally binarizing unit 

„ „ , „ , , . , „ binarizes the gray-scale image within the extracted area. The 

SimJarly, the levek of gray scales of the subpixels 2 „ n h extfaction fa performed by totaling ^ aumber 0 f 

through 4 are obtained as follows. pbK , s having Qne , eve , of ^ ^ of brighlness of aU of 

n-q*rc+(i-q) m [A me P* xe * s °f a document image, as will be described later. If 

the number of pixels is large in a portion where the level of 

I3-q*fD+{Uq)*fB brightness is high, it indicates the background of a document 

15 image. If the number of pixels is large in a portion where the 

/4=pVD4(i-p)vc ] eve l 0 f brightness is low, it indicates the portion including 

AJJV ii .u i i c i t« r.u u * i* a character of the document image. A threshold value is set 

Additionally, the level of gray scale 15 of the subpixel 5 4 . . iL 4 . ... . c • , - ,1 

. , , , • \_ , , . . . . t * i to a value in the portion where the number of pixels is small, 

included m the area enclosed by the original pixels A , , ^ T " . ' ~J , ^ ^^"^ 

4 . , ~ , .... « . 4 ° tL /„ . which corresponds to the middle of the two peaks. This 

through D can be obtained according to the following ... . . . ... 4 lL . c f - 

r ° ° 20 threshold value setting is similar to that of the binarization 

^ a * process. Note that, however, the position at which the 

/5»p^vo+p*(i- 9 )V5+«*(i-p)*/C+(i-p)*(i-^)*M threshold value is set is slightly shifted to a higher level of 

brightness. If the pixels included in a document image are 

The above described calculation is made for all of the extracted with such a threshold value (global threshold 

subpixels arranged between the original pixels, so that the 25 value), the extracted portion will include a character and its 

process for generating subpixels is completed. The obtained periphery. The locally binarizing unit 51 binarizes the image 

levels of gray scale are stored with a method similar to that with another threshold value for the roughly extracted area, 

of original pixel data, along with the corresponding posi- so that noise components caused by the light and shade of 

tions of the subpixels. the background of the document image can be removed. 

FIG. 5 exemplifies a table for specifying the number of 30 Consequently, the binary image which is easier to be rec- 

subpixels to be generated, according to the resolution of a ognized can be obtained. 

gray-scale image which is an input image. Specifically, the global threshold value is determined with 

The process for generating subpixels between original the linear sum of an average pixel value, a standard devia- 

pixels must be performed at the beginning of the calculation tion value, and a variance. Or, it can be determined as 

process explained by referring to FIG. 4. Although the 35 follows. A global process parameter which will appear in the 

number of subpixels to be generated is arbitrary, it must be following equation is a constant, 
set so that sufficient character recognition accuracy can be 

obtained from the image for which subpixels are generated & obal *™k°ld vatucHan average of all pixel valucs^Cglobal 

. ■ . a j- i iL * . , process parameter)* (standard deviation value of all pixel val- 

and which is binanzed. Accordingly, there is no equation for ues) 

uniquely determining the number of subpixels to be gener- 40 

ated according to the resolution of an input image. The The position of the global threshold value is adjusted to 

number of subpixels to be generated must be determined the position where the rough extraction can be most effec- 

based on experimental data to some extent. tively made by making an adjustment with the global 

In FIG. 5, the resolution of ao input image is stored in the process parameter, 

left column in correspondence with a subpixel generation 45 FIG. 7 is a schematic diagram explaining how to set a 

parameter in the right column. For example, if the resolution global threshold value used for the rough extraction process, 

of an input image is 100 dpi, subpixels are generated to form When the rough extraction process is performed, as 

a grid of 6x6 for each original pixel. If the resolution of an shown in this figure, the levels of brightness are obtained 

input image is 150 dpi, subpixels are generated in order to from all of the pixels structuring an entire document image, 

form a grid of 3x3 for each original pixel. 50 and the statistics of the frequency at which a pixel having a 

In this way, the above described subpixel processing unit particular level of brightness appears in the document is 

stores the table shown in FIG. 5. When a gray-scale image collected in a similar manner as in the case where the 

is input, the subpixel processing unit obtains the number of predetermined threshold value is determined for the entire 

subpixels to be generated according to the resolution of the document image. In this figure, the appearance frequency of 

input image, and determines the positions of the subpixels. 5S the pixel against the level of brightness forms a gentle curve. 

Next, the subpixel processing unit obtains the levels of gray However, this is illustrated for ease of explanation about the 

scale of the respective subpixels from the levels of gray scale threshold value setting methoci. If the statistical process 

of the original pixels with the interpolation process by using representing the frequency at which a pixel having a certain 

the calculation method explained by referring to FIG. 4, and level of brightness appears is actually performed with a 

generates the image data having a high resolution according 60 device, the result will be a histogram, 

to the resolution of the input image. In FIG. 7, frequency peaks are formed in the respective 

FIG. 6 is a block diagram exemplifying a further con- portions where the levels of brightness are low and high. The 

figuration of the variable resolution binarizing unit. portion where the level of brightness is high is the back- 

The configuration shown in FIG. 3B relatively requires a ground of a document image, and this level of brightness is 

considerable amount of time in order to locally binarize an 65 the level of brightness of the paper on which the document 

entire image area. In the meantime, with the configuration is created. In the meantime, the portion where the level of 

shown in FIG. 6, a drawing area roughly extracting unit may brightness is low is an area where a character or a graphic, 
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etc. is drawn. If the frequencies at which pixels appear are 
classified depending on the levels of brightness, the pixels 
are grouped into two major groups such as the pixels 
included in the drawing area and the pixels included in a 
background. Therefore, the rough extraction process can be 
performed by setting the threshold value of the level of 
brightness to almost the middle of the two peaks, and 
extracting the pixels having the levels of brightness which 
are lower than this threshold value from the document 
image. 

If a document is structured by the reverse video of a 
black-and-white image, the area where the two peaks are 
drawn and the background become reversed. Accordingly, 
the pixels having levels of brightness which are higher than 
a preset threshold value, are output as a drawing area. 

If an entire document is binarized with a predetermined 
threshold value, the threshold value is set to a value corre- 
sponding to almost the middle of the two peaks, that is, the 
value at the bottom of the frequency curves. This is because 
the character recognition cannot be made with high accuracy 
if the pixels of the background are also captured. When the 
rough extraction process is performed, however, an 
extracted area which includes the periphery of a character is 
more convenient to set a threshold value in the binarization 
process. Therefore, the threshold value is set to the value 
which is slightly shifted from the bottom toward the peak of 
the background. In this way, the area where a character, etc. 
is drawn and its periphery are extracted. 

FIGS. 8A and 8B are block diagrams exemplifying further 
configurations of the variable resolution binarizing unit. 

If the configurations shown in FIGS, 3A and 6 are 
combined, the global process for roughly extracting a draw- 
ing area is performed after the subpixel generation process 
is performed for a gray-scale image. Then, the local bina- 
rization process is performed for each of the pixels included 
in the drawing area. As a result, the processes can be 
performed at a higher speed than that implemented by the 
configuration shown in FIG. 3A, and at the same time, the 
recognition accuracy is improved more than that available 
from the configuration shown in FIG. 6. 

With the configuration shown in FIG. 8 A, the subpixel 
generation process is initially performed by a subpixel 
processing unit 70, and then a document image is roughly 
extracted by a drawing area roughly extracting unit 71. With 
the subpixel generation process, the information about the 
level of brightness of a gray-scale image can be prevented 
from being lost, or rather, the information about the level of 
brightness is included. As a result, the accuracy of the 
binarization process can be improved. Furthermore, per- 
forming the rough extraction process eliminates the need to 
target the whole of a document image, thereby reducing the 
number of pixels to be processed and improving the pro- 
cessing speed of the locally binarizing unit. 

The subpixel processing unit 70 performs the subpixel 
generation process for the whole of a document image. The 
drawing area roughly extracting unit 71 performs the rough 
extraction process by also targeting subpixels. Accordingly, 
the processing speed and the accuracy can be improved 
further than that implemented in the case where only the 
rough extraction process is performed. However, the sub- 
pixel generation process must be performed for the whole of 
the document image in this case. Therefore, the amounts of 
data handled by the subpixel processing unit 70 and the 
drawing area roughly extracting unit 71 increase, which 
leads to the slowdown of the processing speed and the 
insufficiency of the capacity of a memory storing data, 
although the processing speed of the locally binarizing unit 
72 can be improved. 
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FIG. 8B is a block diagram exemplifying the configura- 
tion which improves the processing speed implemented by 
the configuration shown in FIG. 8A. 
The configuration shown in FIG. SB is obtained by 

5 reversing the processing order of the subpixel processing 
unit 70 and the drawing area roughly extracting unit 71, 
which are included in the configuration shown in FIG. 8A. 
Because the process of the drawing area roughly extracting 
unit 73 is performed for the original gray-scale image whose 

10 number of pixels is not increased with the subpixel genera- 
tion process, the amount of data handled by the drawing area 
roughly extracting unit 73 is reduced and the processing 
speed is made faster. 

Additionally, it is sufficient for the subpixel processing 

15 unit 74 to also perform the subpixel generation process, not 
for the whole of an input gray-scale image, but only for the 
area extracted by the drawing area roughly extracting unit 
73. Accordingly, the number of pixels to be handled is 
reduced much more than that in the case where the subpixel 

20 generation process is performed for the whole of the input 
gray-scale image, in the configuration shown in FIG. 8A. 
Consequently, the capacity of the memory storing data can 
be decreased, and at the same time, the processing speed can 
be improved. 

25 Furthermore, since the amount of throughput of the 
locally binarizing unit 75 is approximately the same as that 
of the corresponding unit shown in FIG. 8A, the processing 
speed on the whole can be improved, and also the amount of 
required hardware resources such as a memory, etc. can be 

30 reduced. 

Generally, with the configurations (shown in FIGS. 3 A, 
8A, and 8B) for performing the local binarization process 
after the subpixel generation process, the threshold value of 
the local binarization process is calculated from the subpixel 

35 value at the interpolation point (subpixel). However, the 
subpixel value at the interpolation point is obtained from the 
original pixel values, the local binarization process can be 
performed according to the equation obtained by assigning 
the equation for obtaining a subpixel value at an interpola- 

40 tion point from original pixel values, to the equation for 
obtaining a local threshold value from the subpixel value at 
the interpolation point. 

FIG. 9 is a block diagram exemplifying the details of the 
configuration of the locally binarizing unit. 

45 Because the contents of the subpixel generation process 
performed by a subpixel processing unit 80 are the same as 
those explained by referring to FIG. 4, the explanation is 
omitted here. A locally binarizing unit 81 is composed of a 
threshold value calculating unit 82 and a comparing unit 83. 

50 The threshold value calculating unit 82 directly obtains the 
threshold value of the local binarization process from a 
gray-scale image. The comparing unit 83 makes a compari- 
son between the threshold value and the value at the point 
where a subpixel is generated, which is obtained by the 

55 subpixel processing unit 80, and outputs the result of the 
binarization process. 

That is, the information about a pixel point of an input 
gray-scale image is directly input to the threshold value 
calculating unit 82. Furthermore, the threshold value calcu- 

60 lating unit 82 sets a local area by targeting a particular pixel 
point, statistically classifies the level of gray scale (the level 
of brightness) at each pixel point within the local area as 
explained by referring to FIG. 7, and sets a threshold value 
which allows the distinction between the pixel having the 

65 level of brightness which represents a background, and the 
pixel having the level of brightness which represents a 
drawing area, to be made as definite as possible. The rough 
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extraction process is explained by referring to FIG. 7, and 
the threshold value which is shifted to the level of 
brightness, which represents a background, is described to 
be used. If the binarization process is performed, the thresh- 
old value is set to the value corresponding to the bottom of 
the frequency curves shown in the middle of FIG. 7. With 
such a threshold value setting, the distinction between the 
pixel structuring a character and the pixel structuring a 
background can be definitely made. As a result, a binary 
image which is easier to be recognized can be obtained. 

The threshold value calculated by the threshold value 
calculating unit 82 is input to the comparing unit 83. The 
subpixel processing unit 80 generates the information about 
each pixel point of the gray-scale image for which the 
subpixel generation process is performed, and about the 
level of brightness at each pixel point, and inputs to the 
comparing unit 83 the information about the level of bright- 
ness of the pixel point targeted by the threshold value 
calculating unit 82. The comparing unit 83 then makes a 
comparison between the level of brightness at each pixel 
point and the threshold value. Here, assume that the level of 
brightness at a pixel point, which is higher than the threshold 
value, is set to "0", while the level of brightness at the pixel 
point, which is lower than the threshold value, is set to "1". 
The pixel point to be targeted is sequentially changed, and 
the above described process is repeatedly performed, so that 
a binary image corresponding to the input gray-scale image 
can be obtained. The pixel point referred to in the above 
explanation includes both an original pixel of an input 
gray-scale image and a subpixel generated by performing 
the subpixel generation process. If the pixel point is used to 
indicate an original pixel of an input gray-scale image, it is 
hereinafter referred to as an original pixel point. With the 
configuration shown in FIG. 9, the binarization process is 
performed for the whole of image data whose number of 
pixels is increased by performing the subpixel generation 
process for the whole of the input gray-scale image, 

FIG. 10 shows the details of the second configuration of 
the locally bin arizing unit. 

In this figure, the local threshold value after the subpixel 
generation process is obtained only at each original pixel 
point of a gray-scale image. The local threshold value at an 
interpolation point (a subpixel point) is obtained by inter- 
polating the local threshold values at the original pixel 
points of the gray-scale image. Namely, with the configu- 
ration shown in FIG. 9, both subpixel points and original 
pixel points of a gray-scale image are bandied in a similar 
manner, and the threshold value for the local binarization 
process is calculated. However, with the configuration 
shown in FIG. 10, only the threshold value at an original 
pixel point is calculated, and the threshold value at a 
subpixel point is obtained with the interpolation process. 
Actually, the interpolation process explained by referring to 
FIG. 4 is not performed for the level of brightness at each 
original pixel point. The threshold value at a subpixel point 
is obtained by interpolating threshold values obtained at 
original pixel points, similar to the level of brightness. 

That is, a pixel point threshold value calculating unit 91 
obtains the local threshold value after the subpixel genera- 
tion process only at an original point of a gray-scale image. 
An interpolating unit 92 obtains the local threshold value at 
an interpolation point (subpixel) by interpolating the local 
threshold values at the pixel points of the gray-scale image. 
The comparing unit 83 makes a comparison between the 
pixel value at a pixel point, which is obtained by the subpixel 
processing unit 80, and the threshold value obtained by the 
interpolating unit 92, and outputs the result of the binariza- 
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tion process. Such a bin arizing process is performed for all 
of the pixel points obtained by the subpixel processing unit 
80, thereby obtaining a binary image. 

The method for obtaining the local threshold value at an 
5 original pixel point of a gray-scale image, which is executed 
by the pixel point threshold value calculating unit 91, 
includes the following methods. 

(1) Obtaining the local threshold value at an original pixel 
point of a gray-scale image by using the value of the 

10 subpixel obtained by interpolating original pixel points of 
the gray-scale image. 

(2) Obtaining the local threshold value by substituting the 
equation for obtaining the subpixel value at the interpo- 
lation points from the original pixel values into the 

15 equation for obtaining the local threshold value from the 
subpixel value at the interpolation points. 

(3) Obtaining the subpixel value from the linear combination 
of an average value, a standard deviation value, and a 
variance, by obtaining the standard deviation value and 

20 the variance after obtaining the average value and a 
square average value of values at pixel points of the 
gray-scale image, for which the subpixel generation pro- 
cess is performed, from an equation obtained by substi- 
tuting an equation for obtaining a subpixel value from 

25 original pixel values, to an equation for obtaining the 
average value and the square average value from subpixel 
values. 

The configuration implementing the method (1) is shown 
in FIG. 10. 

30 The configuration implementing the method (2) is shown 
in FIG. U. The pixel point threshold value calculating unit 
uses only the pixel values of a gray-scale image. 

The method (3) is applicable if the local threshold value 
is the linear combination of an average pixel value, a 

35 standard deviation value, and a variance within a beat area. 
The configuration implementing the method (3) is shown in 
FIG. 12. The variance is obtained from the average pixel 
value and the square average value. The standard deviation 
value is obtained from the variance. The specific example of 

40 the equation will be provided next. 

variance-an average of square values-tbe square of an average 
value standard deviation^ V (variance) 

If the local area recognizes original pixels as its boundary, 

45 the equations for obtaining the local threshold value and the 
average values are simplified, which leads to a speeding-up 
of the processing. Therefore, the method using these expres- 
sions may be available. 

Additionally, the locally binarizing unit may comprise a 

50 local area specifying unit, by which the process may be 
performed by making a distinction between the case where 
a local area does not recognize original pixels as its 
boundary, and the case where the local area recognizes the 
original pixels as its boundary (FIG. 13). The local area 

55 specifying unit is a unit for using the number of subpixels 
generated with the interpolation process and the size of a 
local area as its specification data, and a local are an original 
pixel boundary determining unit is a unit for automatically 
determining whether or not the local area recognizes original 

60 pixels as a boundary. 

Provided below are the explanations about the configu- 
rations shown in FIGS. U through 13. 

FIG. 11 exemplifies the third configuration of the locally 
binarizing unit. 

65 The locally binarizing unit 100 is composed of a pixel 
point threshold calculating unit 101, an interpolating unit 
102, and a comparing unit 83. 
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When a gray-scale image is input, the data of the original values are compared with the pixel values transmitted from 
pixels of the gray-scale are input to the subpixel processing the subpixel processing unit 80 in the comparing unit 118, 
unit 80 and the pixel point threshold value calculating unit and the pixel values are then binarized and output. 
101. The subpixel processing unit 80 generates a predeter- FIG* 13 is a block diagram exemplifying the fifth con- 
mined number of subpixels between the original pixels by 5 figuration of the locally binarizing unit, 
performing the above described process. The pixel point Id this example, a locally binarizing unit 120 is composed 
threshold value calculating unit 101 obtains the threshold of a local ™* original pixel boundary determining unit 121, 
value at an original pixel point only from the values of the an onginal pixel boundary pixel point threshold value cal- 
original pixels. The threshold value is obtained by general- c ^ atm S ! * mi } 22 > »f P^l non-boundary pixel point 
• j * u • np n j *u i threshold value calculating unit 123, an interpolating unit 
ing the data shown in FIG. 7, and using the average value, 10 , 
.L * t-u .u u u , . . * ' 124, and a comparing unit 125. 

the variance etc. The threshold value obtained at each ^ da(a Qf £ ^ le ^ are k t t0 the 

original pixel pomt is used by an mterpolatmg unit m order su^i^ssin^^ abo ve described process 

to obtam the threshold value of a subpixel with the method ^ performedt Theil) me data afier me subpixel generation 

such as the linear interpolation method, etc. The comparing process is input to the comparing unit 125. 

unit 83 makes a comparison between the value of an original 15 jh e data 0 f tne input gray-scale image is input also to the 

pixel or the value of a subpixel generated by the subpixel i oca i original pixel boundary determining unit 121 

processing unit, and the threshold value at each pixel point, included in the locally binarizing unit 120. A local area 

which is obtained by the pixel point threshold value calcu- specifying unit 126 is an input unit for specifying where to 

lating unit 101 and the interpolating unit 102, and binarizes set a local area for a particular original pixel. The specifi- 

the pixel value. A binary image is obtained by performing 20 cation may be made either manually or automatically, 

such a process for all of the pixel points. Especially, if the local area is included by the line structuring 

FIG. 12 is a block diagram exemplifying the fourth a character, the local binarization process may be unsuc- 

configuration of the locally binarizing unit. cessfully performed in some cases. It is therefore desirable 

A locally binarizing unit 110 is composed of a pixel point that the local area includes the portion of the line structuring 

threshold value calculating unit 111, an interpolating unit 25 a character and the portion of a background in a ratio of 1 

117, and a comparing unit 118. The contents of the process to «V„ , „ , , . t , , 

performed by the subpixel processing unit 80 are similar to ^ten the range of the local area » set by the local area 

thosedescribedabove.Tnedataofaninputgray^caleimage spf^S "mt 126 for the target pixe 1, the local area 

is input to the subpixel processing una 80, and the pixel on &™ 1 P«el boundary detenninmg unit 121 determines 

• f\u u tj 1 I I*- 111 * i .i j • *u ™ whether or not original pixels exist on the boundary of the 

point threshold value calcula^nig unit 111 included I in tie 30 ^ afca If ^ 0 ^ aal % eb exist on me bonn(h ^ of thc 

locally binarizing unit 110. The pixel pomt threshold value ^ data ^ Emitted to me ori mal pixel boundary 

calculating unit 111 obtains the values at the pixel points pixd ^ threshold value calculating unit 122, which is 

within the local area centering around a targeted pixel point. made to calculate tbe threshold value for each of the original 

The pixel point average value calculating unit 112 calculates pixcls , If mc original pixels do not exist on the boundary of 

an average of the pixel values. A pixel point square average 35 me \ oc& \ ^tn, tne data is transmitted to the original pixel 

value calculating unit 113 calculates an average of the non-boundary pixel point threshold value calculating unit 

squares of the pixel values within thc local area. The average 123, which is made to calculate the threshold value for each 

of the pixel values from the pixel point average value of the original pixels. The methods for calculating a thresh- 

calculating unit 112 and the average of the squares of the old value, which are executed by the original pixel boundary 

pixel values from the pixel point square average value 40 pixel point threshold value calculating unit 122 and the 

calculating unit 113 are input to a pixel point variance original pixel non-boundary pixel point threshold value 

calculating unit 114, which calculates the variance of the calculating unit 123, will be described later. 

distribution of the pixel values. The calculated variance is Since the threshold values calculated by the original pixel 

input to the pixel point standard deviation value calculating point threshold value calculating unit 122 or the original 

unit 15, which obtains the standard deviation value of the 45 pixel non-boundary pixel point threshold value calculating 

distribution of the pixel values. unit 123 are intended only for the original pixels, the 

The average, variance, and standard deviation values of threshold values are transmitted to the interpolating unit 

the pixel values are input to a linear combination calculating 124, which obtains the threshold value for a subpixel by 

unit 116, which calculates a threshold value. The expression performing the interpolation process, 

for calculating a threshold value must be suitably set by a 50 The threshold values obtained for the original pixels and 

skilled artisan. According to this preferred embodiment, the subpixels in this way are transmitted to the comparing 

however, the linear combination of the average and standard unit 125, which makes a comparison between the threshold 

deviation values is used as described above, and the thresh- values and the pixel values transmitted from the subpixel 

old value is adjusted by multiplying the standard deviation processing unit 80. The pixels are then binarized and output, 

value and a parameter in order to adjust to what degree the 55 Here, the expressions for directly calculating a local 

standard deviation value is to affect the threshold value to be average value and a local square average value from original 

set. pixel values will be provided. A local variance and a local 

After the threshold value is obtained for a certain targeted standard deviation value are obtained with fundamental 
pixel point in this way, this threshold value calculation arithmetic operations by calculating the local average value 
process is repeatedly performed for the respective original 60 and the local square average value, and the linear combina- 
pixel points of an input gray-scale image until the threshold tion of the local variance and the local standard deviation 
values are obtained for all of the original pixels. These value is used, whereby a threshold value is obtained accord- 
threshold values are transmitted to the interpolating unit 117, ing to the above described expressions referred to in the 
which then obtains the threshold values of subpixels by explanation of FIGS. 3A and 3B. 

performing the interpolation process. As a result of the 65 FIG. 14 is a schematic diagram explaining the method for 

above described processes, the threshold values of the calculating a local average value and a local square average 

original pixels and the subpixels are obtained. The threshold value from original pixel values. 
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First of all, symbols will be defined. A local area centering 
around an original pixel is illustrated in FIG. 14. The 
coordinate of the targeted original pixel is assumed to be (0, 
0) for ease of explanation. The description of the coordinate 
of a normal original pixel is obtained by parallel translating 5 
the coordinate. The degree of the subpixel generation is 
defined to be n(ill), and (n-1) new subpixels are inserted 
between two original pixels. The local area is a square area 
centering around a targeted original pixel. The upper left 
point (that is, the coordinate of the upper left original pixel) J(J 
of the regular square, which is the original pixel farthest 
from the targeted original point, among the original pixels 
included in the local area, is assumed to be (-M, -M) with 
the condition l^M imposed. The number of subpixels, 
which exist outside the regular square but within the local 
area in one direction, after the subpixel generation process 15 
is assumed to be "r". "r" must satisfy 0^r<n. The original 
pixel value at a coordinate (i, j) is represented as I(i, j). 

Assuming that one side of the local area is "L", the 
following equation is satisfied. 

20 

Z,=2(Af*K+r)+l (O^rcn) 

That is, "M" original pixels center around the targeted 
pixel point, for example, in a vertical direction, "n-1" 
subpixels exist between the original pixels, and "r" subpix- 2S 
els exist outside the "M" original pixels. Since the pixels 
exist in both of the top and the bottom of the local area in 
a similar manner, 2(M*n+r) is obtained. Additionally, the 
single target point is added, so that the above described 
equation is obtained. 

"L" is an odd number larger than "2n*\ Conversely, if "L" 30 
which is an odd number larger than a positive number "n", 
and "2n" are given, "M" and "r" which satisfy the above 
described equation are uniquely determined. That is, if 
(L-l)/2 is divided by "n" and its quotient and remainder are 
obtained, the quotient and the remainder respectively cor- 35 
respond to "M" and V. 

Represented below are the equations for obtaining a local 
average value and a local square average value, which are 
respectively used in the case (1) where the boundary of a 
local area is original pixels (r=0), and in a normal case (2). 40 
The expressions used in the case (1) are simpler than those 
used in the case (2). The expressions are represented as 
follows. 

local average value "£"(/) 

45 

local square average valued (V 2 ) 

Q(i~0, 1, 2, 3, . . . ) is a coefficient which will appear in 
the following expressions. Even if identical symbols are 
used, their definitions are different depending on the expres- 50 
sions. Since E(I) and Efl 2 ) are multiplied by C 0 in the 
following expressions, it is easily understood that the right 
side of the expressions is divided by c a in order to obtain E(I) 
and E(I 2 ). 

(1) In the case where original pixels exist on the boundary 5S 
of a local area (r»0) 

Note that the coefficients are defined by the 



following equations 

C 2 -2«(n+l) 

C Q E(I 2 ) = 



M-l M-l 



m — 1 i m — 1 m — i 

Ci Z Z / ( i *j> 2+c 2 Z < / (- M **) 2+ 

i=-M+l /a-M+1 i=-M+i 

/(W, kf 4- /(A, - A/) 2 + /(*, A/) 2 } ¥ C 3 {/(-M, -M f +■ 



/(- M. M) 2 +■ l(M, -M? + /(M, M) 2 ) 



M-l M-l 

c < Z Z '(wWw + i) + 

i=-M+l j=-M 
M-l 

Cs £ m-M. J + l) + /(M. j + [)} 

j=-u 

( 



C 4 £ M-l) £ /<*. /)'(*+ l,/> + 
r=-M J=-M+l 

M-l 

C 5 £ -M)/(i+ 1, -M) + /(/, M)/(i + 1, M)) 



i--M 
M-l M-l 



C «Z Z WwW+l.j+U + 

i=-M y=-M 



/(i.y+l)/(l+l,»} 



Note that the coefficients are defined by the following 
equations. 

(2) In a normal case 



C 0 -36(2Mn+l) 2 /i 2 
C^/i+l) 2 ^!) 2 



C 4 =4<* 2 -l)(2w 2 +l) 

C 5 -2(»+lXnM)(2/H-l) 

C^n 2 -]) 2 
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;=-w+i }=~m*i i=-*+i 
i(M. k) + /(*. -M) + !{k, M)} + C 3 t/(-M, -A/) + 
/(-W, M ) + !{M, -M) + i(M, M)) + 

M-l 

C 4 £ l'<~M -1.*) + AM +!.*) + /(*. -M-l) + 

i— M+l 

/(A, M + 1)} + C 5 \f(-M - 1, -M) +■ I(-M - 1. M) + 
/(M+ U -M)+/(M +1, M) + /(-M, -M - L) + 
/(-M, M + 1)+ /(W, -M - 1)+ /(M, M + 1)} + 
C 6 tf(-M - 1, ~M, -1) + t(-M - 1. M + I) + 
/(M + 1, -M - 1) + f{M +■ 1, Af + 
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15 



Note that the coefficients are defined by the following 
equations. 

C 0 =4(2Afr>+l)V 
C t =4n 4 

C 2 -2n 2 {n 2 +(2r+fyt-r{r+i)} 

C 3 =n 4 +2(2^+l)n 3 ^2r I +2r+l>t 2 -2Krfl)(2r+l>H.r 2 (r+l 
C 4 »2r(rfl)rt 2 

C 5 -r<M-l){Q 2 +(2r+l>i-r(rfl)} 

Note that the coefficients are defined by the following 
equations. 



M-l W-l M-l 20 

C 0 E(l 2 )= C, 2 J] /(/wf + C 2 J] |/(-Af , A) 2 4- 

J=-«+l Jt=-M+1 

/(Af , A) 2 -I- /(A, -Af) 2 + /(A, M?) + C 3 {/(-A* , -Af ) 3 4- 
/(-Af, Af) 2 4-/(Af, -Af) 2 4-/(Af, Af) 2 }4- ^ 

M-l 

C 4 Z Ut-Af-l.A^-h/tAf 4-I,A) 2 4- 
A=-JW+l 

/(jt t _Af _ l) 2 + t(k, Af 4- if] + CjJ/(- Af - 1 , -Af ) 2 + 
/(_Af _ i f Af) 2 + /{Af 4- 1, - Af) 2 4- I(M + 1, Af) 2 4- 30 
/(-Af , -Af - l) 2 4- /(-Af , Af 4- 1) 2 4- /(Af , -Af - l) 2 4- 
/(Af , Af 4- l) 2 } 4- C c tf(-Af - 1, -Af - I) 2 4- 
/(-Af - 1, Af + if + /(Af 4- 1, -Af - if 4- /(Af 4- 1, Af +- 35 

M-l M-l 

c ? Z Z /(/,/)/a>+i)+ 

l-Mtl y— « 
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-continued 

M-l 

C 9 Yj U C ~M - + 1. - Af - 1) 4- 

/(i,Af 4-l)/(/-Hl, Af 4-l)H- 
V-l 

do X W-W-l./)/(-Af,/) + 

/(AY, /)/(Af + 1, J)} 4- CjH/(-Af - 1, -Af)/(-Af , -Af) + 

/(Af , -Af)/(Af 4- 1 T - Af) 4- /(-Af - 1 T Af )/(-Af, Af) + 

/(Af , M)/(Af + 1, Af )j + 

C«i/(-M - 1, - Af - l)/(-Af, -Af - 1)4- 

/(Af . -A/ - l)/(Af +• J, -Af - 1)4- 

/(-Af - 1, Af 4- 1)/(- Af , Af + 1) 4- 



/(Af.Af 4-l)/(Af + 1. Af 4-1)} 



M-l M-l 

C "Z Z Cft/W+l,/+l)t 

!)/</+ 1./)} + 
C i4 £ (W-W-l i ft)/(-M,A+l) + 

*=-« 

/(_Af - 1, A 4- l)/{- Af , A)} 4- {/(Af , A)/(Af 4- 1, A + I) + 
/(Af,A + l)/(Af 4-KA)}4-{/(A. -Af - 1)/(A 4- 1, -Af ) 4- 
/(A4-I,-Af-1)/(A,-Af)}4- 

i/(A, Af)/(A 4- 1, Af + l)L/(* + 1. M)I{-M, - Af - 1)})4- 
C ls ({/(-Af - 1, - Af - l)/(-Af , -Af) + 
/(-Af - 1. - Af )/(-Af , - Af - 1)) + 
U(-Af - 1, Af )/(- Af , Af 4- 1) 4- 
/(-Af-1, Af + !)/(- Af, Af)}4- 
/(Af.-Af -l)/(Af 4-1. + 
/(Af , -Af )/(Af 4- 1, -Af - 1)} 4- 
U(Af , Af)/(Af 4- 1, Af 4- 1) 4- /(Af , Af 4- l)/(Af 4- 1, Af }) 



C 9 £ V(-Af-l,/)/(-Af-l, ;+l)4. 

j=-M 

/(Af 4-1, ;)/(Af 4- 1, y+l)}4- 45 

The expression which is used in the case where "M" is 
Cio Z * /( '* _M _ 1 W» _Af) + ^ M + 1)1 + equal t0 or larger 11110 ^ re P resented for tDe 

i— Af*l 

CiH/(-Af , -Af - l)/(- Af , -Af) + 
/(-Af , Af)/(-Af, Af + 1) 4- /(Af, - Af - l)/(Af . -Af)4- 
/(Af, Af)/(Af, Af4-1)}+- 
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/(-Af -1. Af)/(-Af-l,Af + 1)4- 
/(Af 4- 1, -Af - l)/(Af 4- 1, -Af ) + 
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C 0 -36» 4 (2AfW+2M-l) 2 
C!»4« 2 (2« 2 +l) 2 

C 2 -2«{4n 3 +6(2r+l)a 4 4(-3r 2 -3/4-l)a 5 4-(4f :> 4.6/ 2 +Sr4.3)n 2 +(-6/ 2 - 
6r4-l>»4-7<r*-l)(2rt.l)} 

C 3 M« 6 4-12(2r4-l)n 5 4-(12^4.12M.13)Q 4 -2(32r 5 +48/ 2 4-10r-3)n 3 

4-(60rVl20r , +54r 2 -6r+l)n 2 -2^12rS30^4-22r J 4-3r-l>t+^ 
l) 2 (2r+l) 2 

C 4 -2r<rH)(2r+l>t(2rt 2 4-l) 

C 5 =<r4.1)(2r4-l){2n 5 4-3C2«.l)n 2 4-(-6^-6rfl>+r(r+l)(2rl)} 
C 6 -^(^l) 2 (2r+l) 2 

C 9 =2^r4-l)(2r+l>i(rt 2 -l) 
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C l6 -4<r+l)»(3/i-C2j-+l) )(2/i 2 +l) 

C l2 -2^r+l) 2 (2r+l)(3rt-(2r+l) 

C 14 =2<r+l>(« 2 -l)(3/i-(2r+l) 
C 1J -2r z («-l) 2 (3rt-(2rt.l) 

normal case (2). The expression which is used in the case 
where M»0 can be obtained in the following three steps. 

(1) Setting "M" to "1". 

(2) Replacing "n" with "r" 

(3) Replacing the value of "F* with the boundary value of 
a local area 

The variance and standard deviation values can be 
obtained by using the average value and the square average 
value of the pixel values within a local area, which are 
obtained with the above described expressions, thereby 
obtaining the expression for determining a threshold value. 

FIG. 15 is a block diagram exemplifying the sixth con- 
figuration of the locally binarizing unit. 

With this configuration, the value obtained by interpolat- 
ing the difference between the pixel value at an original pixel 
point of a gray-scale image and the local threshold value 
after the subpixel generation process, at an interpolation 
point (subpixel), and the value of the binary image at the 
interpolation point, are determined with the sign of the 
interpolation value. Because the interpolation process is 
reduced from twice to once with this configuration, the 
processing speed can be improved. Especially, if the sub- 
pixel generation process is performed with the linear inter- 
polation method, the same effects as those in the case where 
respective original pixels are interpolated and compared can 
be obtained. 

the subpixel value for {original pixel value-binarization threshold 
value }= the value obtained by performing the subpixel genera- 
tion process for original pixel values- the value obtained by 
performing the subpixel generation process for a binarization 
threshold value 

That is, the locally binarizing unit 140 is composed of a 
pixel point threshold value calculating unit 140, a difference 
calculating unit 142, an interpolating unit 143, and a sign 
determining unit 144. 

The data of an input gray-scale image are directly input to 
the difference calculating unit 142, and to the pixel point 
threshold value calculating unit 141. The pixel point thresh- 
old value calculating unit 141 calculates the threshold value 
within a local area only from the original pixel values 
included in the local area for the original pixel point of the 
input gray-scale image, and generates the threshold value for 
a targeted original pixel. Next, the targeted original pixel 
value and the generated threshold value are input to the 
difference calculating unit 142, which calculates the differ- 
ence between them. The value of a subpixel is then obtained 
by interpolating this difference in a similar manner as in the 
above described process for interpolating the level of bright- 
ness (gray scale). Since the obtained value is the same as the 
value of the difference between the value of the subpixel and 
the threshold value, the difference from the value of the 
original pixel and that of the subpixel are input to the sign 
determining unit 144, which examines the sign of the 
difference. 

For example, if the pixel value is larger than the threshold 
value, the above described difference becomes positive, and, 
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for example, the value "0" is assigned to this pixel when 
being binarized. If the pixel value is smaller than the 
threshold value, the difference becomes negative, and, for 
example, the value "1" is assigned to the pixel when being 
5 binarized. By performing such a process for all of the pixel 
points, a binary image can be obtained. 

FIG. 16 is a block diagram exemplifying the seventh 
configuration of the local binarizing unit. 

With this configuration, a binary image is obtained by 
lQ obtaining the local threshold value after the subpixel gen- 
eration process at each pixel point of a gray-scale image, and 
by using the table to which the values at 4 pixel points of the 
gray-scale image and the local threshold value and from 
which the binary image enclosed by 4 pixel points is output 
as data. 

15 Namely, a locally binarizing unit ISO is composed of a 
pixel point threshold value calculating unit 151, a binary 
image searching unit 152, and a memory 153. The pixel 
point threshold value calculating unit 151 obtains the thresh- 
old value in a local area from the pixel values of the input 

20 gray-scale image for each of the pixels, and provides it to the 
binary image searching unit 152. The binary image search- 
ing unit 152 receives the threshold values from the pixel 
point threshold value calculating unit 151 and the original 
pixel values of the gray-scale image, and selects 4 original 

25 pixels forming the regular square which is a minimum unit 
of the grid formed by original pixel points. Then, a binary 
image is obtained by referencing the table stored in the 
memory 153 based on the pixel values and the threshold 
values of the 4 original pixels. The table stored in the 

30 memory 153 is a table to which the binary image data is 
registered for the combination of the pixel values and the 
threshold values of 4 original pixels. The binary image 
searching unit 152 obtains the binary image data for all of 
unit grids (minimum regular squares forming a grid) struc- 

35 tured by the original pixels of the input gray -scale image 
according to this table, generates an entire binary image by 
combining the data, and outputs the generated image. 

Only the explanations about the configurations of the 
variable resolution binarizing unit were provided above. 

40 However, as shown in FIG. 6, the rough extraction process 
for roughly extracting a drawing area is initially performed 
for a gray-scale image, and the above described process is 
performed only for the extracted drawing area, thereby 
reducing the amount of processing time. 

45 FIG. 17 snows the processing examples up to the process 
for binarizing a color image, and FIG. 18 shows a gray-scale 
image according to this preferred embodiment. 

FIG. 17 shows the processing example of a 150-dpi color 
image. 

50 The top image is a color image document with a resolu- 
tion of 150 dpi. In an actual color image, various colors 
appear around a black color representing characters. The top 
color image converted into a gray-scale image is shown as 
a middle image. The gray-scale image binarized with the 

55 conventional method is the binary image on the right at the 
bottom. Viewing this image, the detailed portions of the 
characters are defaced and the characters are difficult to 
recognize. In the meantime, the binary image obtained by 
performing the processing according to this preferred 

60 embodiment is the image on the left at the bottom. Since 
subpixels are generated according to this preferred 
embodiment, the resolution of the binary image is substan- 
tially higher than that of the original color image. In this 
case, the amount of information increases. However, since 

65 the binary image data is used not for printing by a printer, 
etc, but for charter recognition as it is, the clearer represen- 
tation of the characters facilitates the character recognition. 
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FIG. 18 shows the processing example of 150- and 
100-dpi gray-scale images. 

If the 150- and 100-dpi gray-scale images are binarized by 
a conventional process, an erroneous character recognition 
result may often be obtained due to the defacement of the 5 
characters. In the meantime, according to this preferred 
embodiment, subpixels are generated and the information 
about the levels of brightness can be prevented from being 
lost, whereby the characters can be recognized more clearly, 
and a higher recognition rate can be realized when the 10 
character recognition process is performed. 

FIG. 19 is a block diagram explaining the configuration of 
the hardware required for implementing this preferred 
embodiment as software. 

In this preferred embodiment, the subpixel generation 1S 
process, the interpolation process, the threshold value cal- 
culation process, etc. can be implemented as a program 
running on a computer. In this case, as the hardware con- 
figuration required for the computer, a CPU 181 for per- 
forming the above described processes, a RAM 183 for 20 
storing the program for performing these processes in an 
executable form, etc. must be interconnected by a bus 180, 
so that they can communicate with each other. Furthermore, 
a ROM 182 for storing the BIOS required for running the 
CPU 181, a storage device 189 for storing the program, etc. 2S 
are arranged. The storage device 189 is implemented, for 
example, as a hard disk, etc. In addition, a storage medium 
reading device 187 is required when the program is stored 
onto a portable storage medium 188 such as a floppy disk, 
a CD-ROM, etc. and used. The program read from the 30 
storage device 189 or the portable storage medium 188 is 
expanded and stored in the RAM 183 so that the CPU 181 
can execute it. Furthermore, an input/output device 186 
composed of a monitor, a keyboard, a mouse, etc. is arranged 
in order to transmit to the CPU 181 the commands issued by 3S 
a user who operates the device, and to display the results of 
the processes performed by the CPU 181 for the user. 

Furthermore, the program may not be stored in a com- 
puter used by a user. It may be used depending on need by 
being downloaded from a database possessed by a program ^ 
provider 185. Or, the program may be executed in a network 
by connecting the user and the program provider 185 via a 
LAN. Only a command input and a result display are 
performed by the computer possessed by the user in this 
case. 45 

As described above in detail, according to the present 
invention, a color or gray-scale document image can be 
quickly binarized with high accuracy, thereby recognizing 
the image accurately and rapidly. 

What is claimed is: 50 

1. A document image recognizing device, comprising: 

image converting means for converting an input docu- 
ment image into a gray-scale image if the input docu- 
ment image is a color image, and for newly outputting 
a gray-scale image if the input image is a gray-scale 55 
image; 

variable resolution binarizing means for converting the 
input document image into a binary image having a 
higher resolution according to a resolution of the gray- 
scale image; wherein said variable resolution binariz- 60 
ing means performs a sub-pixel generation process for 
increasing a number of pixels included in an image by 
interpolating pixel values of a gray-scale image, sets a 
local threshold value within a local area centering 
around a particular pixel, and obtains a binary image by 65 
using the local threshold value; and 

recognizing means for recognizing the binarized image. 
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2. The device according to claim 1, wherein: 

said recognizing means recognizes a converted or input 
binary image, and converts the binary image into 
electronic codes. 

3. The device according to claim 1, further comprising: 
drawing area roughly extracting means for roughly 

extracting a drawing area according to a global thresh- 
old value for pixel values of a gray-scale image, 
wherein 

said recognizing means recognizes a binary image in an 
area extracted by said drawing area roughly extracting 
means. 

4. The device according to claim 3, wherein: 

the global threshold value uses a linear combination of an 
average pixel value, a standard deviation value, and a 
variance. 

5. The device according to claim 3, wherein 

said variable resolution binarizing means performs the 
subpixel generation process for an entire input gray- 
scale image; and 

said drawing area roughly extracting means roughly 
extracts a drawing area from gray-scale image data for 
which the subpixel generation process is performed. 

6. The device according to claim 1, wherein: 

the subpixel generation process is performed by linearly 
interpolating original pixel values of the gray-scale 
image. 

7. The device according to claim 1, wherein: 

said variable resolution binarizing means obtains a binary 
image by binarizing pixel values by using the local 
threshold value obtained from a distribution of the pixel 
values within a local area including pixels; and 

the local threshold value uses a linear combination of an 
average pixel value, a standard deviation value, and a 
variance. 

8. The device according to claim 1, further comprising: 
drawing area roughly extracting means for performing a 

global process which roughly extracts a drawing area 
according to a global threshold value for pixel values of 
a gray-scale image, wherein 
said variable resolution binarizing means performs the 
subpixel generation process for increasing a number of 
pixels included in an image by interpolating pixel 
values of a gray-scale image, for a drawing area which 
is roughly extracted with the global process, and per- 
forms a local binarization process by using the local 
threshold value for each pixel included in the roughly 
extracted drawing area. 

9. The device according to claim 1, wherein the local 
threshold value at a pixel point after the subpixel generation 
process is based on obtaining a value of a subpixel generated 
with the subpixel generation process from original pixel 
values, and obtaining a local threshold value from subpixel 
values. 

10. The device according to claim 1, wherein a local 
threshold value after the subpixel generation process is 
obtained at a pixel point of a gray-scale image, and a local 
threshold value of a subpixel is obtained by interpolating 
local threshold values at pixel points of the gray-scale 
image. 

11. The device according to claim 10, wherein the local 
threshold value at the pixel point of the gray-scale image is 
obtained by using a value of a subpixel obtained by per- 
forming the subpixel generation process for pixel points of 
the gray-scale image. 
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12. The device according to claim 10, wherein the local 
threshold value at the pixel point of the gray-scale image 
after the subpixel generation process is based on obtaining 
a subpixel value from original pixel values, and obtaining a 
local threshold value from subpixel values. 5 

13. The device according to claim 12, wherein the local 
threshold value is obtained, using subpixel values, from a 
local threshold value from a local area that recognizes 
original pixels as its boundary. 

14. The device according to claim 10, wherein an inter- 10 
polation value at a subpixel point is obtained by interpolat- 
ing a difference between a value at a pixel point of a 
gray-scale image and a local threshold value after the 
subpixel generation process, and a value of a binary image 

at the subpixel point is determined with a sign of the 15 
interpolation value. 

15. The device according to claim 1, wherein a local 
threshold value at an original pixel point of a gray-scale 
image after the subpixel generation process is obtained from 

a linear combination of an average value, a standard devia- 20 
tion value, and a variance, by obtaining the standard devia- 
tion value and the variance after obtaining the average value 
and a square average value of values at pixel points of the 
gray-scale image, for which the subpixel generation process 
is performed, based on obtaining a subpixel value from 25 
original pixel values, and obtaining the average value and 
the square average value from subpixel values. 

16. The device according to claim 1, wherein: 

the local area recognizes original pixels as its boundary; 
and 30 

a local threshold value at an original pixel point of a 
gray-scale image after the subpixel generation process 
is obtained from a linear combination of an average 
value, a standard deviation value, and a variance, by 
obtaining the standard deviation value and the variance 35 
after obtaining the average value and a square average 
value of values at pixel points of the gray -scale image, 
for which the subpixel generation process is performed, 
based on obtaining a subpixel value from original pixel 
values, and obtaining the average value and the square 40 
average value from subpixel values. 

17. The device according to claim 1, further comprising: 
specifying means for specifying a range of a local area, 

wherein 

45 

whether or not the local area recognizes original pixels as 
its boundary is determined by using as specification 
data a number of subpixels generated with the subpixel 
generation process and a size of the local area. 

18. The device according to claim 1, wherein: 5Q 
a local threshold value after the subpixel generation 

process is obtained at a pixel point of a gray-scale 
image; 

a table to which pixel values and local threshold values at 
four pixel points of a gray-scale are input and from 55 
which a binary image enclosed by the four pixel points 
is output as data, is included; and 

a binary image is obtained by using said table. 

19. A document image recognizing method, comprising: 
converting an input document image into a gray-scale 60 

image if the input document image is a color image, 
and newly outputting a gray-scale image if the input 
image is a gray-scale image; 
converting the input document image into a binary image 
having a higher resolution according to a resolution of 65 
the gray-scale image; wherein said converting the input 
document image into a binary image performs a sub- 
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pixel generation process for increasing a number of 
pixels included in an image by interpolating pixel 
values of a gray-scale image, sets a local threshold 
value within a local area centering around a particular 
pixel, and obtains a binary image by using the local 
threshold value; and 
recognizing the binary image. 

20. The method according to claim 19, wherein: 

said recognizing the binary image recognizes a converted 
or input binary image, and converts the binary image 
into electronic codes. 

21. The method according to claim 20, further compris- 
ing: 

roughly extracting a drawing area according to a global 
threshold value for pixel values of a gray-scale image, 
wherein 

said recognizing the binary image recognizes a binary 
image in an area extracted by said roughly extracting a 
drawing area. 

22. The method according to claim 21, wherein the global 
threshold value uses a linear combination of an average 
pixel value, a standard deviation value, and a variance. 

23. The method according to claim 21, wherein: 

said converting the input document image into a binary 
image performs the subpixel generation process for an 
entire input gray-scale image; and 

said roughly extracting a drawing area roughly extracts a 
drawing area from gray-scale image data for which the 
subpixel generation process is performed. 

24. The method according to claim 19, wherein: 

the subpixel generation process is performed by linearly 
interpolating original pixel values of the gray-scale 
image. 

25. The method according to claim 19, wherein: 

said converting the input document image into a binary 
image obtains a binary image by bin arizing pixel values 
by using the local threshold value obtained from a 
distribution of the pixel values within a local area 
including pixels; and 

the local threshold value uses a linear combination of an 
average pixel value, a standard deviation value, and a 
variance. 

26. The method according to claim 19, further compris- 
ing: 

performing a global process which roughly extracts a 
drawing area according to a global threshold value for 
pixel values of a gray-scale image, wherein 

said converting the input document image into a binary 
image performs the subpixel generation process for 
increasing a number of pixels included in an image by 
interpolating pixel values of a gray-scale image, for a 
drawing area which is roughly extracted with the global 
process, and performs a local binarization process by 
using the local threshold value for each pixel included 
in the roughly extracted drawing area. 

27. The method according to claim 19, wherein the local 
threshold value at a pixel point after the subpixel generation 
process is based on obtaining a value of a subpixel generated 
with the subpixel generation process from original pixel 
values, and obtaining a local threshold value from subpixel 
values. 

28. The method according to claim 19, wherein a local 
threshold value after the subpixel generation process is 
obtained at a pixel point of a gray-scale image, and a local 
threshold value of a subpixel is obtained by interpolating 
local threshold values at pixel points of the gray-scale 
image. 
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29. The method according to claim 28, wherein the local 
threshold value at the pixel point of the gray-scale image 
after the subpixel generation process is based on obtaining 
a subpixel value from original pixel values, and obtaining a 
local threshold value from subpixel values. 

30. The method according to claim 29, wherein the local 
threshold value is obtained, using subpixel values, from a 
local threshold value within a local area that recognizes 
original pixels as its boundary. 

31. The method according to claim 28, wherein an inter- 
polation value at a subpixel point is obtained by interpolat- 
ing a difference between a value at a pixel point of a 
gray-scale image and a local threshold value after the 
subpixel generation process, and a value of a binary image 
at the subpixel point is determined by a sign of the inter- 
polation value. 

32. The method according to claim 19, wherein the local 
threshold value at the pixel point of the gray-scale image is 
obtained by using a value of a subpixel obtained by per- 
forming the subpixel generation process for pixel points of 
the gray-scale image. 

33. The method according to claim 19, wherein a local 
threshold value at an original pixel point of a gray-scale 
image after the subpixel generation process is obtained from 
a linear combination of an average value, a standard devia- 
tion value, and a variance, by obtaining the standard devia- 
tion value and the variance after obtaining the average value 
and a square average value of values at pixel points of the 
gray-scale image, for which the subpixel generation process 
is performed, based on obtaining a subpixel value from 
original pixel values, and obtaining the average value and 
the square average value from subpixel values. 

34. The method according to claim 19, wherein: 

the local area recognizes original pixels as its boundary; 
and 

a local threshold value at an original pixel point of a 
gray-scale image after the subpixel generation process 
is obtained from a linear combination of an average 
value, a standard deviation value, and a variance, by 
obtaining the standard deviation value and the variance 
after obtaining the average value and a square average 
value of values at pixel points of the gray-scale image, 
for which the subpixel generation process is performed, 
based on obtaining a subpixel value from original pixel 
values, and obtaining the average value and the square 
average value from subpixel values. 

35. The method according to claim 19, further compris- 
ing: 

specifying a range of a local area, wherein whether or not 
the local area recognizes original pixels as its boundary 
is determined by using as specification data a number 
of subpixels generated with the subpixel generation 
process and a size of the local area. 

36. The method according to claim 19, wherein: 

a local threshold value after the subpixel generation 
process is obtained at a pixel point of a gray-scale 
image; 

a table to which pixel values and local threshold values at 
four pixel points of a gray-scale are input and from 
which a binary image enclosed by the four pixel points 
is output as data, is included; and 

a binary image is obtained by using said table. 

37. A computer-readable storage medium for directing a 
computer to execute a process comprising: 

converting an input document image into a gray-scale 
image if the input document image is a color image, 



and newly outputting a gray-scale image if the input 
image is a gray-scale image; 
converting the input document image into a binary image 
having a higher resolution according to a resolution of 

5 a gray-scale image; wherein said converting the input 
document image into a binary image performs a sub- 
pixel generation process for increasing a number of 
pixels included in an image by interpolating pixel 
values of a gray-scale image, sets a local threshold 

10 value within a local area centering around a particular 
pixel, and obtains a binary image by using the local 
threshold value; and 
recognizing the binary image. 

38. The storage medium according to claim 37, wherein: 
said recognizing the binary image recognizes a converted 

or input binary image, and converts the binary image 
into electronic codes. 

39. The storage medium according to claim 37, wherein 
20 the process further comprises: 

roughly extracting a drawing area according to a global 
threshold value for pixel values of a gray-scale image, 
and wherein 

said recognizing the binary image recognizes a binary 
25 image in an area extracted by said roughly extracting a 
drawing area. 

40. The storage medium according to claim 39, wherein 
the global threshold value uses a linear combination of an 
average pixel value, a standard deviation value, and a 

30 variance. 

41. The storage medium according to claim 39, wherein: 
said converting the input document image into a binary 

image performs the subpixel generation process for an 
entire input gray-scale image; and 
35 said roughly extracting a drawing area roughly extracts a 
drawing area from gray-scale image data for which the 
subpixel generation process is performed. 

42. The storage medium according to claim 39, wherein 
the subpixel generation process is performed by linearly 
interpolating original pixel values of the gray-scale image. 

43. The storage medium according to claim 37, wherein: 
said converting the input document image into a binary 

image obtains a binary image by binarizing pixel values 
by using the local threshold value obtained from a 
distribution of the pixel values within a local area 
including pixels; and 
the local threshold value uses a linear combination of an 
average pixel value, a standard deviation value, and a 
50 variance. 

44. The storage medium according to claim 37, further 
comprising: 

performing a global process which roughly extracts a 
drawing area according to a global threshold value for 

55 pixel values of a gray-scale image, and wherein 

said converting the input document image into a binary 
image performs the subpixel generation process for 
increasing a number of pixels included in an image by 
interpolating pixel values of a gray-scale image, for a 

60 drawing area which is roughly extracted with the global 
process, and performs a local binarization process by 
using the local threshold value for each pixel included 
in the roughly extracted drawing area. 

45. The storage medium according to claim 37, wherein 
65 the local threshold value at a pixel point after the subpixel 

generation process is based on obtaining a value of a 
subpixel generated with the subpixel generation process 
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from original pixel values, and obtaining a local threshold 
value from subpixel values. 

46. The storage medium according to claim 37, wherein 
a local threshold value after the subpixel generation process 

is obtained at a pixel point of a gray-scale image, and a local 5 
threshold value of a subpixel is obtained by interpolating 
local threshold values at pixel points of the gray-scale 
image. 

47. The storage medium according to claim 46, wherein 
the local threshold value at the pixel point of the gray-scale 10 
image is obtained by using a value of a subpixel obtained by 
performing the subpixel generation process for pixel points 

of the gray-scale image. 

48. The storage medium according to claim 46, wherein 
the local threshold value at the pixel point of the gray-scale 15 
image after the subpixel generation process is based on 
obtaining a subpixel value from original pixel values, and 
obtaining a local threshold value from subpixel values. 

49. The storage medium according to claim 48, wherein 
the local threshold value is obtained, using subpixel values, 20 
from a local threshold value within a local area that recog- 
nizes original pixels as its boundary. 

50. The storage medium according to claim 46, wherein 
an interpolation value at a subpixel point is obtained by 
interpolating a difference between a value at a pixel point of 25 
a gray-scale image and a local threshold value after the 
subpixel generation process, and a value of a binary image 

at the subpixel point is determined with a sign of the 
interpolation value. 

51. The storage medium according to claim 37, wherein 30 
a local threshold value at an original pixel point of a 
gray-scale image after the subpixel generation process is 
obtained from a linear combination of an average value, a 
standard deviation value, and a variance, by obtaining the 
standard deviation value and the variance after obtaining the 35 
average value and a square average value of values at pixel 
points of the gray-scale image, for which the subpixel 
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generation process is performed, based on obtaining a sub- 
pixel value from original pixel values, and obtaining the 
average value and the square average value from subpixel 
values. 

52. The storage medium according to claim 37, wherein: 
the local area recognizes original pixels as its boundary; 

and 

a local threshold value at an original pixel point of a 
gray-scale image after the subpixel generation process 
is obtained from a linear combination of an average 
value, a standard deviation value, and a variance, by 
obtaining the standard deviation value and the variance 
after obtaining the average value and a square average 
value of values at pixel points of the gray-scale image, 
for which the subpixel generation process is performed, 
based on obtaining a subpixel value from original pixel 
values, and obtaining the average value and the square 
average value from subpixel values. 

53. The storage medium according to claim 37, further 
comprising: 

specifying a range of a local area, and wherein 
whether or not the local area recognizes original pixels as 
its boundary is determined by using as specification 
data a number of subpixels generated with the subpixel 
generation process and a size of the local area. 

54. The storage medium according to claim 37, wherein: 
a local threshold value after the subpixel generation 

process is obtained at a pixel point of a gray-scale 
image; 

a table to which pixel values and local threshold values at 
four pixel points of a gray -scale are input and from 
which a binary image enclosed by the four pixel points 
is output as data, is included; and 

a binary image is obtained by using said table. 

***** 
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ABSTRACT 



A visual bar code recognition method which combines 
conventional decoding techniques with optical character 
recognition (OCR). The visual bar code recognition method 
captures an image of an object containing a bar code. 
Regardless of the orientation of the bar code within the\ 
field-of-view, the system detects the presence of the barl 
code, and decodes it using the bar/space patterns. It thenf 
produces an independent decoding of the human-readable 
numbers printed on the bar code using OCR. From these two 
decodings, it determines the identity of the object. It verifies 
this identity by comparing the physical characteristics of the 
object from the image with the known features of the 
product. 

14 Claims, 5 Drawing Sheets 
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VISUAL BAR CODE RECOGNITION 
METHOD 

BACKGROUND OF THE INVENTION 

The present invention relates to image processing meth- 
ods and bar code scanners, and more specifically to a visual 
bar code recognition method. 

Bar code symbols provide a fast and accurate means of 
representing information about an object. Decoding or read- 
ing of the bar code is accomplished by translating the 
patterns of bars and spaces into a unique series of numbers 
that correspond to a specific item. Currently, the majority of 
bar codes are read using laser scanners. While these laser 
systems work well under optimal conditions, they have 
some inherent disadvantages and limitations. For example, a 
laser scanner cannot verify that a correct bar code has been 
scanned by an examination of the item. 

Therefore, it would be desirable to provide a visual bar 
code recognition method, which may be part of a visual bar 
code scanner. 

SUMMARY OF THE INVENTION 

In accordance with the teachings of the present invention, 
a visual bar code recognition method is provided. 

The visual bar code recognition method captures a color 
image of an object containing a bar code. Regardless of the 
orientation of the bar code within the field-of-view, the 
system detects the presence of the bar code, and decodes it 
using the bar/space patterns. It then produces an independent 
decoding of the human-readable numbers printed on the bar 
code using OCR. From these two decodings, it determines 
the identity of the object. It verifies this identity by com- 
paring the physical characteristics of the object from the 
image with the known features of the product. 

The method adds a great deal of decoding flexibility not 
possible with a laser since a laser scanner can only perform 
a subset of these tasks. 

As long as the bar code is within the focused field of view, 
the algorithms are capable of locating and decoding the bar 
code regardless of orientation. 

Error detection and correction are provided in two ways: 
(a) Multiple scan lines are passed through the bar code 
allowing for partial decoding of several lines and recom- 
bining into a final result and (b) Optical Character Recog- 
nition (OCR) on the characters below the bar code normally 
used for manual entry allow for an independent decoding. 

After decoding, products may be verified by comparing 
the known physical characteristics (color, size, shape, 
texture, etc.) of the decoded product with the features found 
in the captured image. 

The method is applicable to both processing on a single 
frame static image (such as with a hand-held camera) or on 
a real-time image stream. The implementations of these 
ideas are different in each case but the underlying process is 
identical. 

It is accordingly an object of the present invention to 
provide a visual bar code recognition method. 

It is another object of the present invention to provide a 
visual bar code recognition method which uses a camera to 
capture an image of an item having a bar code. 

It is another object of the present invention to provide a 
visual bar code recognition method which uses a camera to 
capture an image of an item having a bar code, and which 
locates and decodes the bar code. 



56,696 Bl 

2 

It is another object of the present invention to provide a 
visual bar code recognition method which uses a camera to 
capture an image of an item having a bar code, and which 
verifies the contents of the bar code using optical character 
s recognition (OCR). 

It is another object of the present invention to provide a 
visual bar code recognition method which uses a camera to 
capture an image of an item having a bar code, and which 
verifies the contents of the bar code by comparing features 
io of the item from the image with features in a product 
database. 

It is another object of the present invention to provide a 
visual bar code recognition method which uses a camera to 
capture an image of an item having a bar code which may 
15 be static or moving. 

BRIEF DESCRIPTION OF THE DRAWING 

Additional benefits and advantages of the present inven- 
tion will become apparent to those skilled in the art to which 
20 this invention relates from the subsequent description of the 
preferred embodiments and the appended claims, taken in 
conjunction with the accompanying drawings, in which: 
FIG. 1 is a block diagram of a visual bar code system of 
^ the present invention; 

FIG. 2 is a block diagram of the bar code reading program 
of FIG. 1; 

FIG. 3 is a block diagram of the bar code location and 
orientation module of FIG. 2; 
30 FIG. 4 is a block diagram of the scan line extraction and 
decoding module of FIG. 2; 

FIG. 5 is a block diagram of the optical character recog- 
nition (OCR) decoding module of FIG. 2; 

FIG. 6 is a block diagram of the combination and veri- 
35 fication module of FIG. 2; and 

FIGS. 7A and 7B form a flow diagram of the visual bar 
code decoding method. 

DETAILED DESCRIPTION OF THE 
40 PREFERRED EMBODIMENT 

Referring now to FIG. 1, system 10 primarily includes 
camera 14, terminal 16, and transaction server 17. 

Camera 14 produces an image 28 of bar code 24 and item 
26. Preferably, image 28 is a 24-bit color image. Bar code 24 
includes black and white bars and human-readable charac- 
ters. Item 26 may be static or moving. 

Terminal 16 includes processor 12. 

Processor 12 executes transaction processing software 21 
50 which tallies items during a transaction, including item 26. 
In order to obtain price information for an item, transaction 
processing software 21 sends a price request containing an 
item number obtained from bar code reading software 22 to 
transaction server 17. 
55 Bar code reading software 22 locates and decodes bar 
code 24 and sends the item number to transaction processing 
software 21. Bar code reading software 22 produces decoded 
bar code information by analyzing image 28, by using 
optical character recognition of numeric characters printed 
60 with the bar code and evident in image 28, and by comparing 
features extracted from image 28 with features stored within 
product database 30. 

Transaction server 17 provides item price and item 
descriptions from PLU file 33 in response to requests from 
65 terminal 16. 

Storage medium 18 permanently stores bar code reading 
software 22 and contains product database 30. Product 
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database 30 contains item features that bar code reading 
software 22 uses to identify item 26. Thus, only items whose 
features have been previously entered in product database 30 
are identifiable. 

Display 20 displays item price and item descriptions 32 S 
provided by processor 12. 

Storage medium 31 stores PLU file 33. 

Turning now to FIG. 2, bar code reading software 22 
includes bar code location and orientation determining mod- 
ule 34, scan line extraction and decoding module 36, optical 10 
character recognition (OCR) decoding module 38, and com- 
bination and verification module 40. 

Bar code location and orientation determining module 34 
determines the location 42 and orientation 44 of bar code 24 
and produces a gray scale image 46 from color image 28. 15 
Bar code location and orientation determining module 34 
utilizes the highly parallel nature of the bars within bar code 
24. In an image such as image 28, the edges or boundaries 
of these parallel bars will themselves be parallel, pointing in 
a direction perpendicular to the bars. The presence of a 20 
compact set of unidirectional edges signals the possibility of 
a bar code. In frequency space, the predominance of a single 
phase of these edges filters a bar code from surrounding text 
and determines the orientation of the bar code. This edge- 
based approach to bar code location and orientation is valid 25 
in both a static image (such as one produced using a 
hand -held camera) or a frame from an real-time image 
stream. 

Scan line extraction and decoding module 36 uses loca- 
tion 42, orientation 44, and gray scale image 46 to decode 30 
bar code 24 from its bar and space patterns and produces 
decoded bar code characters string 48. By-products of scan 
line decoding are bar code type 50, bar code direction 52, 
and binary image 54. 

OCR decoding module 38 uses bar code type 50, bar code 
direction 52, and binary image 54, along with bar code 
location 42 and orientation 44, to extract the precise regions 
in image 46 that contain the human-readable characters of 
bar code 28 and produce decoded human-readable charac- 
ters string 56. 

Combination and verification module 40 produces a final 
decoding 58 of bar code 24 from three sources: decoded bar 
code characters string 48, decoded human-readable charac- 
ter string 56, and features in image 28. 45 

Turning now to FIG. 3, bar code location and orientation 
determining module 34 includes color to gray scale conver- 
sion module 60, edge detection module 62, magnitude 
analysis module 64, and phase analysis module 66. 

Color to gray scale conversion module 60 transforms 50 
image 28 into an 8 -bit per pixel representation, which is gray 
scale image 46. 

Edge detection module 62 applies a filter at each pixel of 
gray scale image 46 to produce a gray scale edge map 68 that 
indicates whether each point of image 46 is a member of the 55 
set of bar code edge pixels (the stronger the edge in the 
image, the greater the gray level in edge map 68.) The filter 
filters out pixels having gray scale levels below a predeter- 
mined threshold gray scale level. 

Magnitude analysis module 64 analyzes edge map 68 to 60 
provide the location of bar code 24, An area of edge map 68 
with a dense concentration of high strength edges indicates 
a region of good contrast and the likely location of bar code 
24. Magnitude analysis module 64 looks for a density of 
pixels left from the filtering by edge detection module 62 65 
that is greater than a predetermined threshold density. The 
center of the region is termed the location 42 of bar code 24. 
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Phase analysis module 66 employs location information 
42 from magnitude analysis module 64 and edge map 68 to 
determine the orientation 44 of bar code 24. If the phase or 
direction is similar for most of the high strength edges, the 
region probably contains the parallel bars of bar code 24. 
The phase of the edges is the bar code's orientation 44. 

Referring now to FIG. 4, scan line extraction and decod- 
ing module 36 includes bar code traversing module 70, 
threshold selection algorithm 72, image thresholding mod- 
ule 74, scan line correlation and enumeration module 76, 
and bar code decoding module 78. 

Bar code traversing module 70 traces a series of gray scale 
scan lines 80 completely across the bar code 24 at the 
computed orientation angle 44. This bar code traversal is 
comprised of three steps: defining coordinates that make up 
a line at the given angle (with origin at (0,0)), determining 
starting points at both sides of the bar code (since its 
direction is unknown at this stage), and creating a set of scan 
lines 80 using each starting point (from Step 2) as an offset 
to the line determined in Step 1. 

Due to the design of the bar code itself, it is not necessary 
for this angle 44 to be exact. All that is required is that each 
scan line 80 passes completely through the bar code 24. The 
coordinates in Step 1 may be computed as needed or 
predetermined and stored in a lookup table. 

Threshold selection algorithm 72 transforms the gray 
scale image 46 into a binary image 54. This is accomplished 
by selecting a single value or threshold and mapping all 
pixels whose gray levels are greater than the threshold to one 
and all those below the threshold to zero. 

The literature supports a large number of threshold selec- 
tion algorithms 72 and the results of this bar code decoding 
method are not dependent on any particular method. 
Experimentally, numerical techniques appear to be prefer- 
able to statistical (histogram) methods. To allow for a margin 
of error when thresholding, it may be desirable to select 
more than one threshold (usually by varying parameters in 
a single thresholding scheme). The threshold used to create 
the binary image 54 is either the single computed value or 
the average of several. 

Imaging thresholding module 74 transforms gray scale 
scan lines 80. If N thresholds are selected, the thresholded 
output contains N+l levels. Therefore, the output of imaging 
thresholding module 74 may be of binary, ternary, or higher 
order. 

Scan line correlation and enumeration module 76 pro- 
duces a complete set of binary scan lines 84. The correlation 
part of this step combines groups of M adjacent lines in a 
effort to reduce the effects of noise and thresholding arti- 
facts. From these combined scan lines, the enumerator 
portion constructs a set of binary scan lines 84 representing 
all possible pixel patterns. 

Bar code decoding module 78 decodes each of scan lines 
84 by measuring the bar/space patterns and translating them 
into a string 48 containing the bar code characters along with 
the bar code type 50 (UPC- A, UPC-E, etc.) and the bar code 
direction 52 (left to right or right to left). 

Bar code decoding module 78 additionally verifies that 
each decoded string 48 satisfies the checksum requirements 
for a bar code. If errors are found in one portion of a scan 
line, bar code decoding module 78 will attempt to salvage 
any information possible from sections of the line. Ideally, 
all of the scan lines will decode to the same bar code. If not, 
bar code decoding module 78 then assigns probabilities to 
the different decodings based on the number of scan lines 
producing each. The algorithms developed to decode laser 
scanned bar codes are applicable here. 
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With reference to FIG. 5, OCR decoding module 38 concentration of high-strength edges is not found, then item 

includes image derotation module 90, subimage extraction 26 is not properly oriented or printed or does not have a bar 

module 92, and OCR module 94. code. The method proceeds to step 164, in which software 

Subimage extractor 92 appropriately locates the subim- 22 displays an error message on display 20 and the method 

ages 98 of binary image 54 that contain the printed human- 5 ends m ste P 166. 

readable characters. For instance, UPC-A has ten characters, If image 46 contains a dense concentration of high- 
separated into two groups of five, printed directly below the strength edges, the method proceeds to step 124. 
bars (inside the guard bars), one on the bottom left and one In step 124, phase analysis module 66 determines whether 
on the bottom right. Subimage extractor 92 places rectangles the edges have a similar phase so that the area found in step 
at these four locations and isolates the pixels within these 10 12 0 may be classified as a bar code. 
k° xes * If not, then the area is not a bar code or bar code 24 is so 

OCR module 94 then reads the characters from subimages poor i y printed that decoding is not possible. The method 

98, choosing only numbers as possible characters. It then proceeds to step 164, in which software 22 displays an error 

combines the decodings from all subimages 98 into a single message on display 20 and the method ends in step 166. 

bar code string 56, again validating the checksum. 15 [f ^ edges haye a simUar ^ ^ area ^ ^ 

Image derotation module 90 produces a binary image 96 a bar ^de and the method proceeds to step 126. 

in which bar code 24 is guaranteed to be vertical This is Irj stc m hase ^ ^ module 66 dctermincs oricn . 

necessary if OCR module 94 cannot handle rotated text in tation ^ of ^ar code 24 

subimages 98. Rotated subimages 100 are passed through „„„ . ," . , f „_ 

OCR module 94 to image derotation module 90, as dt 20 . f n US > b " 00(16 * ave ™ n 8 module 70 P roduces 

cussed further below. individual gray scale scan hnes 80. 

If a single character decodes as more than one number, ln S, <P 130 ' threshold adec,ion d g° ri,hm 72 P rovides 

both selections are retained and probabilities are assigned to res 0 

each. Decoding of such numbers is resolved by combination In ste P 132 > ima S e thresholding module 74 produces 

and verification module 40. thresholded scan lines 82. 

With reference to FIG. 6, combination and verification Inste P 134 ' scan line correlation and enumeration module 

module 40 includes combination decoding module 102 and 7 . 6 P roduces binarv ima S e 54 and a complete set of binary 

product verification module 104. ^ ncs ** 4, 

Combination decoding module 102 compares strings 48 30 In ste P 136 * bar code decodiri g module 78 attempts to 

and 56 to determine an estimate 106 of the identity of item decode bm ^ hnes 84 

26. If there are errors, the method proceeds to step 164, in 

Product verification module 104 compares features iden- which software 22 displays an error message on display 20 

dried within image 28 with features stored within product and the method ends in ste P 166 - 

database 30 to estimate the identity of item 26. Features 35 If there are no errors, then the method proceeds to step 

include, but are not limited to, such attributes as shape, size, 138. 

and color scheme of item packaging, and text and logos In step 138, bar code decoding module 78 produces bar 

printed on the item or the item packaging. code characters, barcode type 50, and bar code direction 52. 

Since both the bar code string 48 and the OCR string 56 In step 140, bar code decoding module 78 determines 

may have a degree of uncertainty associated with each 40 whether the characters represent a single bar code, 

character, product verification module 104 then compares If so, the method proceeds to step 144. 

estimate 106 suggested by strings 48 and 56 with the If not) me meta od proceeds to step 142. 

estimate determined by product verification module 104. fa U2 ^ ^ decodi mQdule ?g 

Uncertainty may be caused J>y one or more cha^i5 being abimies tQ ±e decoded and ^ ihc \^ d 

undecodable using OCR decoding module 38 or by the 4S ^ ^ babilitie J 0 form stri ^ 

bar/space patterns being undecodable. Therefore, it is . 

entirely possible that neither method of modules 36 and 38 . In ste P 144 ' subl 1 ma S e extractor module 92 ^ates sub " 

can decode bar code 24 correctly. Product verification mod- ima f s 9 * tha ! ™ known to contam mc P nntcd characters 

ule 104 produces the final decoding 58 of item 26 based on ot bar code 24 * 

these probabilities. The final decoding 56 is a series of 50 In sle P 146 > subimage extractor module 92 determines 

numbers that corresponds to item 26 in the product database whether characters within the subimages are rotated. 

30. If not, the method proceeds to step 145. 

TAirning now to FIGS. 7 A and 7B, the method of operation If so, the method proceeds to step 148. 

of system 10 and software 22 is illustrated beginning with In step 148, subimage extractor module 92 passes the 

START 110. 55 rotated subimage 100 to image derotation module 90. 

In step 112, camera 14 produces 24-bit color image 28 of In step 150, image derotation module 90 produces a 

bar code 24 and item 26. Item 26 may be static or moving vertical binary image 96 and the method returns to step 144 

in front of camera 14. until all of the subimages are derotated. 

In step 114, color to gray scale conversion module 60 60 In step 145, OCR module 94 combines the decodings 

transforms image 28 into an 8-bit per pixel gray scale image from all subimages 98 through OCR module 94 into string 

46. This step decreases memory and processing power that 56. 

are required to locate bar code 24. i n slep 152, OCR module 94 determines whether string 56 

In step 116, edge detection module 62 produces gray scale represents a valid bar code, 

edge map 68. 65 If not, the method proceeds to step 164, in which software 

In step 120, magnitude analysis module 64 attempts to 22 displays an error message on display 20 and the method 

locate bar code 24. If an area characterized by a dense ends in step 166. 
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If so, then the method proceeds to step 154. 

In step 154, combination decoding module 102 compares 
string 48 with string 56 to produce a series of numbers to 
form estimate 106. 

In step 156, product verification module 104 compares 
features identified within image 28 with features stored 
within product database 30 to determine whether estimate 
106 is in product database 30. 

If not, the method proceeds to step 164, in which software 
22 displays an error message on display 20 and the method 
ends in step 166. 

If so, then the method proceeds to step 160. 

In step 160, transaction software 21 sends a request to 
transaction server 17 to retrieve the price of item 26 from 
PLU file 33. Transaction software 21 adds the item and its 
price to the transaction total and completes the transaction 
after all such items have been processed by software 22. 

The method ends in step 166. 

Although the present invention has been described with 
particular reference to certain preferred embodiments 
thereof, variations and modifications of the present inven- 
tion can be effected within the spirit and scope of the 
following claims. 

What in claimed is: 

1. A method of decoding a bar code to identify an item 
having the bar code on it comprising the steps of: 

capturing an image of the item; 

locating the bar code label in an area of the image; 

decoding the bar code label located in the area of the 

image to produce a first set of characters; 
performing optical character recognition of the area of the 

image to produce a second set of characters; and 
comparing the first set of characters to the second set of 

characters to identify the item. 

2. The method as recited in claim 1, further comprising 
the steps of: 

determining predetermined features of the item from the 
image; and 

comparing the features to features stored within a data- 
base to verify the identity of the item. 

3. A method of decoding a bar code on an item comprising 
the steps of: 

providing a gray-scale image of the item; 

determining gray levels of pixels within the image; 

filtering out pixels having gray levels below a predeter- 
mined threshold gray level; 

determining an area of the image having a density of 
pixels with gray levels above the predetermined thresh- 
old density to locate the bar code; 

determining an orientation of the area; 

tracing a plurality of gray-scale lines through the area; 

transforming the image into a binary image; 

transforming the gray-scale lines to binary lines; 

decoding the binary lines to produce a first set of char- 
acters; 

performing optical character recognition of the area to 

produce a second set of characters; 
comparing the first set of characters to the second set of 

characters to identify the item; 
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determining predetermined features of the item from the 
image; and comparing the features to features stored 
within a database to verify the identity of the item. 

4. The method of claim 1 wherein the step of capturing an 
s image of the item is performed utilizing a camera to scan a 

substantial portion of the item including the bar code. 

5. The method of claim 1 wherein the step of locating the 
bar code label further comprises the steps of determining the 
location of a dense concentration of high-strength edges 

50 within the image and analyzing whether said edges have a 
similar phase. 

6. The method of claim 1 wherein the bar code has an 
associated human readable character string located in close 

15 proximity to a series of bars and spaces defining a coded 
portion of the bar code, said step of decoding the bar code 
label comprises a decoding of the series of bars and spaces, 
and said step of performing optical character recognition 
comprises a recognition of said associated human readable 

20 character string. 

7. The method of claim 6 wherein identification of the 
item occurs only if the first set of characters and the second 
set of characters sufficiently match. 

8. The method of claim 2 wherein said predetermined 
25 features correspond to physical characteristics of the item. 

9. The method of claim 8 wherein identification of the 
item occurs only if the first set of characters and the second 
set of characters sufficiently match and said predetermined 
features sufficiently match the features stored within the 

30 database. 

10. A visual recognition apparatus for decoding a bar code 
to identify an item, the apparatus comprising: 

a camera for capturing an image of the item; 

35 a bar code label locator for locating an area within the 
image including the bar code; 
a decoder for decoding the bar code and to produce a first 
set of characters corresponding to the bar code; 

40 an optical character recognizer for recognizing a human- 
readable string of characters associated with the bar 
code and also located in said area, and for producing a 
second set of characters corresponding to the human- 
readable string of characters; and 

45 a processor for comparing the first set of characters and 
the second set of characters and to identify the item if 
the first set of characters and the second set of charac- 
ters sufficiently match. 

11. The apparatus of claim 10 wherein the processor is 
50 further operable to determine predetermined features of the 

item from the image, and to compare the predetermined 
features to features stored within a database to further verify 
the identity of the item. 

12. The apparatus of claim 10 wherein the bar code label 
55 locator further comprises an edge filter for locating an area 

within the image having a dense concentration of high- 
strength edges. 

13. The apparatus of claim 12 further comprising an 
analyzer for analyzing whether said edges have a similar 

60 phase. 

14. The apparatus of claim 11 wherein the predetermined 
features correspond to physical features of the item. 

* * * * * 
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[57] ABSTRACT 

A test and alignment system for an electronic display device 
comprises a test pattern generator to be connected to an 
electronic display device for causing images of video test 
patterns to be displayed by the electronic display device. A 
test fixture is positioned in front of the electronic display 
device to be tested and aligned. The test fixture includes a 
frame supporting a plurality of close-up optical sensors to 
sense and produce image signals corresponding to small 
areas of images displayed on the electronic display device 
and a plurality of wide-angle optical sensors behind the 
close-up optical sensors for sensing and producing image 
signals corresponding to large areas of images displayed on 
the electronic display device. A computer controls the test 
pattern generator and processes and analyses the image 
signals generated by the close-up and wide-angle optical 
sensors to perform a series of tests on the electronic display 
device. A display provides a visual indication of the results 
of the series of tests performed by the computer. 

37 Claims, 10 Drawing Sheets 
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TEST AND ALIGNMENT SYSTEM FOR and alignment functions include color purity, focus, static 

ELECTRONIC DISPLAY DEVICES AND and dynamic convergence, yolk rotation, orthogonality, 

TEST FIXTURE FOR SAME video pattern size and centering, raster size and centering, 

linearity, geometry and luminance measurements and are all 

RELATED APPLICATIONS 5 performed at a single station. During the series of test and 

„ ... - nm alignment functions, the computer displays graphical indi- 

The present Whcation, a m^aUra-in?art of PCT cators which are used b of , 0 detclmiat tf the 

application senal No PCT/CA95/0(G52 filed on Jm 13, electronic dis , device y passes P or fails a lest . 

1995 design aUng the United States which is a continuation* ITT _ ,. . XT ... , , XT 

in-part of U.S. application Ser. No. 08/259,309 filed on Jun. lft . . " L? 3 ^ «P?lM»tion No. 2,255 700 pubbshed on Nov. 

13, 1994, now abandoned. 10 U ' ^ , P d ^ ncd t0 Sa ?" u W Hectron l > ev J 0 « Co °°- 

pany Ltd. discloses a system for measuring cathode ray tube 

FIELD OF THE INVENTION (CRT) characteristics. The system comprises a camera, a 

selector, a video processor, a monitor, a magnetic field 

The present invention relates to testing systems and in controller, a CPU and an output unit. The camera includes an 

particular to a test and alignment system for an electronic 15 array of CCD cameras as well as a plurality of magnetic field 

display device and to a test fixture for the same. coils which are energized by the magnetic field controller 

under control of the CPU to move an electron beam within 

BACKGROUND OF THE INVENTION the CRT. Image output of the CCD cameras is conveyed to 

During the manufacture and assembly of electronic dis- , n CPU for P**™**- * ° nc embodiment, the CCD 

play devices, such as for example cathode ray tube (CRT) 20 cam f ras ™ m0UDtcd °? t a nil™ a* 

c * •* ■ . i ■ * , ' position of the CCD cameras relative to the CRT under 

assemblies for computer monitors and television sets, pre- ^ : , , 

, . , r , * * . . * ■ . , exaimnation to be adjusted, 

cise mechanical, optical and electronic adjustments are J 

required to ensure the electronic display devices provide Although the abovendescribed references show test and 

optimum reproduction image quality. These adjustments „ alignment systems for electronic display devices including a 

include, but are not limited to, focus, purity of color, plurality of CCD cameras which are moveable relative to the 

convergence of beams, color uniformity, geometry, and electronic display device under exaimnation and allow at 

luminance. Each of these adjustments is typically performed least one characteristic of the electronic display dev lC e to be 

independently of the others by a trained technician with the measured, improved systems for testing and aligning elec- 

aid of a testing and alignment system. Unfortunately, many 3Q ^c display devices are continually being sought. 

testing and alignment systems are deficient because they are It is therefore an object of the present invention to provide 

unable to perform all of the tests needed to align precisely a novel test and alignment system for an electronic display 

a CRT assembly, are very slow and unable to provide device and a test fixture for the same. 

real-time feedback for operator adjustments, are not sufE- cn ..,, Anv „ TK n/irNrn™j 

* *i * j * * j* _c u r*u SUMMARY OF THE INVENTION 

cicntly accurate and are not integrated to perform each of the 35 

tests in an optimized sequence and then recheck and readjust According to one aspect of the present invention there is 

measurements as required. provided a test fixture for a test and alignment system for an 

To deal with the above-described problems, improved test electronic display device comprising: 

and alignment systems for electronic display devices have a frame; 

been developed. For example, PCT application serial No. 40 a plurality of close-up optical sensors mounted on said 
PCT/CA95/00352 published on Dec. 21, 1995 and assigned frame to sense and produce image signals correspond- 
to Image Processing Systems Inc., the assignee of the ing to small areas of an image displayed on an elec- 
present invention, discloses a test and alignment system for tronic display device positioned adjacent said test fix- 
electronic display devices. The system includes a plurality of ture; and 

color CCD cameras mounted on a frame and arranged in an 45 a plurality of wide-angle optical sensors mounted on said 
array. The rows of CCD cameras in the array are vertically frame behind said close-up optical sensors, said wide- 
adjustable and the CCD cameras in each row are laterally angle optical sensors for sensing and producing image 
adjustable to allow the system to test and align different signals corresponding to large areas of said image 
sized electronic display devices. The image output of the displayed on said electronic display device. 
CCD cameras is conveyed to a computer for processing. The 50 Preferably, the wide-angle optical sensors have fields of 
computer controls a test pattern generator connected to the v j ew which include the small areas of the displayed image 
electronic display device being tested so that the electronic anc j which overlap so that the wide-angle optical sensors 
display device displays appropriate test patterns of which sense and produce image signals corresponding substan- 
images are captured by the CCD cameras. A wobulator is tiaily to the entire image displayed on the electronic display 
mounted on the frame and is moveable between an operable 55 device. It is also preferred that the close-up optical sensors 
position adjacent the electronic display device and a are positioned on the frame so as to be outside of the 
retracted position outside of the fields of view of the CCD overlapping fields of view of the wide-angle optical sensors, 
cameras. The wobulator is energizable to bend electron \ Q onc embodiment, the close-up optical sensors are 
beams within the electronic display device when in the arranged in an array with the rows of close-up optical 
operable position. Photodiodes, also mounted on the frame, 60 sensors in the array being mounted on vertically adjustable, 
measure the luminance of the electronic display device as angled rails. The optical sensors in each row are also 
the wobulator bends the electron beams therein and provide moveable laterally along the rails to allow the position of the 
input to the computer for processing. optical sensors to be adjusted to accommodate different 
The computer performs a comprehensive series of test and sized electronic display devices. Preferably, the array 
alignment functions on the electronic display device based 65 includes a centrally positioned, vertically adjustable close- 
on the image output of the CCD cameras and the output of up optical sensor mounted on a cantilever extending from 
the photodiodes as the test patterns are displayed. The test the frame. 
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It is also preferred that the test fixture includes at least one 
wobulator mounted on the frame for creating a magnetic 
field to bend electron beams within the electronic display 
device. In one embodiment, the wobulator includes at least 
one pair of spaced coils mounted on the frame which are 5 
energizable to create an alternating magnetic field sufficient 
to encompass generally the entire electronic display device. 
It is however preferred that the wobulator includes a pair of 
vertically spaced, generally horizontal coils and a pair of 
laterally spaced, generally vertical coils with each of the 10 
pairs of coils being energizable to create the alternating 
magnetic fields and being fixed to the frame in positions 
outside of the field of view of the close-up and wide-angle 
optical sensors. 

According to another aspect of the present invention there 15 
is provided a test and alignment system for an electronic 
display device comprising: 

a test pattern generator to be connected to an electronic 
display device for causing said electronic display 
device to generate images of video test patterns; 20 
a test fixture including a frame supporting a plurality of 
close-up optical sensors to sense and produce image 
signals corresponding to small areas of images dis- 
played on said electronic display device and a plurality 
of wide-angle optical sensors behind said close-up 25 
optical sensors for sensing and producing image signals 
corresponding to large areas of images displayed on 
said electronic display device; 
a computer controlling said test pattern generator and 
processing said image signals to perform a series of 
tests on said electronic display device; and 
a display to provide a visual indication of the results of 

said series of tests. 
According to still yet another aspect of the present inven- 35 
tion there is provided a test fixture for a test and alignment 
system for an electronic display device comprising: 
a frame; 

at least one optical sensor mounted on said frame to sense 
and produce image signals corresponding to an image 40 
displayed on an electronic display device positioned 
adjacent said test fixture; and 
at least one wobulator mounted on said frame and includ- 
ing at least one pair of spaced coils energizable to 
create an alternating magnetic field to bend electron 45 
beams within said electronic display device, said coils 
being fixed to said frame outside of the field of view of 
said at least one optical sensor. 
The present invention provides advantages in that the 
close-up optical sensors in combination with the wide-angle 50 
optical sensors mounted on the frame allow small areas and 
large areas of images displayed on the electronic display 
device to be captured simultaneously without requiring the 
close-up optical sensors to be moved. This allows a com- 
prehensive series of test and alignment functions to be 55 
performed quickly on the electronic display device. Also, the 
wobulator design allows the coils to be permanently 
mounted on the frame outside of the fields of view of the 
close-up and wide-angle optical sensors while still allowing 
it to generate the desired alternating magnetic fields to bend 60 
electron beams within the electronic display device. This 
further speeds up the test and alignment procedure. 

BRIEF DESCRIPTION OF THE DRAWINGS 

An embodiment of the present invention will now be 65 
described more fully with reference to the accompanying 
drawings in which: 
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FIG. 1 is a block diagram of a test and alignment system 
for an electronic display device in accordance with the 
present invention; 

FIG. 2a is a perspective view of a test fixture forming part 
of the test and alignment system illustrated in FIG. 1; 

FIG. 2b is a top cross-sectional view of the test fixture 
illustrated in FIG. 2a; 

FIG. 2c is a side cross-sectional view of the test fixture 
illustrated in FIG. 2a; 

FIG. 3 is an enlarged cross^sectional view of a portion of 
the test fixture illustrated in FIG. 2a; 

FIG. 4 is an enlarged cross-sectional view of another 
portion of the test fixture illustrated in FIG. 2a; 

FIGS. 5 to 10 are diagrams illustrating graphical panels 
displayed to an operator during operation of the test and 
alignment system illustrated in FIG. 1; and 

FIG. 11 is a front elevational view of a calibration grid 
used to calibrate wide-angle cameras forming part of the test 
and alignment system illustrated in FIG. 1. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

Referring to FIG. 1, a schematic illustration of a test and 
alignment system for an electronic display device 12 includ- 
ing CRT assemblies and television sets or computer moni- 
tors is shown and is generally indicated to by reference 
numeral 10. System 10 includes a test fixture 14 on which 
are mounted a plurality of optical sensors 16 to capture 
images of test patterns displayed on the electronic display 
device 12 and to generate video image signals corresponding 
to the captured images. A wobulator 18 is also mounted on 
the test fixture 14 to bend electron beams within the elec- 
tronic display device 12 as will be described. 

The video image signal output of the optical sensors 16 is 
conveyed to a computer 20 by way of a multiplexer and high 
speed video input channel processor generally indicated to 
by reference numeral 22. The computer 20 is connected to 
an operator display 24, to a test pattern generator 26 and to 
the wobulator 18. The test pattern generator 26 feeds test 
patterns to the electronic display device 12 for display under 
the control of the computer 20. The computer 20 can also be 
connected to a programmable logic controller (PLC) 28 for 
controlling movement of a conventional conveyor belt 30, 
on which electronic display devices are supported, to posi- 
tion each electronic display device to be tested in front of the 
test fixture 14. A power supply including deflection ampli- 
fiers and video amplifiers, heater and high voltage power 
sources, commonly referred to as a scan unit 32, that is 
required to operate the electronic display device 12 is also 
shown for reference. 

The computer 20 analyses the video image signal output 
of the optical sensors 16 after being processed by the 
multiplexer and video input channel processor 22 and per- 
forms a comprehensive series of tests and alignment func- 
tions to determine the condition of the electronic display 
device 12. The test and alignment functions performed by 
the computer 20 include color purity, focus, static and 
dynamic convergence, yoke rotation, perpendicularity or 
orthogonality, video pattern size and centering, raster size 
and centering, linearity, geometry, and luminance measure- 
ments including brightness cut-off. During testing, the com- 
puter 20 generates graphical indicators which are displayed 
on the operator display 24 to allow the operator to determine 
easily if the electronic display device 12 passes or fails each 
test. 



12/17/2003, EAST Version: 1.4.1 



5,969,756 

5 6 

Referring now to FIGS. 2a to 2c, the test fixture 14 is displayed on the electronic display device 12 is in focus 

better illustrated. As can be seen, the test fixture 14 includes when it strikes the CCD chip. The CCD chip 112 is mounted 

a box- like frame 50 which accommodates the optical sensors on a sliding assembly 114, which is. able to reciprocate in a 

16. The optical sensors 16 are arranged in two sets and linear direction parallel to the optical axis 116 through lens 

include an array 52 of close-up digital color CCD cameras 5 110 - The sliding assembly 114 is abutted against and guided 

54 and an array 56 of wide-angle digital color CCD cameras b y a Unear slide element 118 > ^ "* driven b y a molor 120 

58. In this particular example shown, array 52 includes three via a lead-screw assembly 122. The motor 120 is responsive 

rows with each row having three CCD cameras 54 and array ^ motor controller 68 to position the CCD chip 112 in 

56 includes two rows with each row having two CCD relation to the lens 110 so that the image displayed on the 

cameras 58. Those of skill in the art will however appreciate 10 5^?/°™ de J lce U A and P r0 J ected onto CCD cm P 

that the number of CCD cameras shown is for illustrative 112 b * lens 110 15 shar P and * focus - 

purposes only and that the test fixture 14 may include a Referring again to FIGS. 2a to 2c, mounted on the face 

significantly larger number of CCD cameras 54 and 58 P lale 76 and surrounding the cantilever 74 is the array 56 of 

arranged in larger arrays 52 and 56. wide-angle CCD cameras 58. The wide-angle CCD cameras 

™. - A . , . , is 58 have overlapping fields of view so that the video image 

The CCD cameras 54 in the upper and lower rows of array 15 . t 4 r* L i eo . 

„ j . rt 7~ , ,/ signal output of the wide-angle CCD cameras 58 represents 

52 are supported by carnages 60 mounted on generally . & . „ K 4 . , . . . V,. , 

horizontal r^ls 62. The raik 62 extend between a pair of 6DUre ° D ^ e,ectromcd,s P la y 

generally vertical rails 64 fixed to the frame 50. The ends of m . _ ( mA . , 

the rails 62 are received by drives 66 in the form of ^ ***** onenution of the CCD cameras 54 in the 

servo-motors. The drives 66 are movable along the vertical 20 «PP« and lower rows of array 52 and the positions of the 

rails 64 in response to input from a motor controller 68 (see CCD ^meras 54 supported on the stubs 70 and cantilever 74 

FIG. 1), operated by the computer 20 to adjust the vertical resuU m l } e C ? D b ° ag ° f ?l °u 

position of the upper and lower rows of CCD cameras 54. ™ wo j the *? de ;!^ le CCD c ^ etas 58 ™"f' ev * D lh ° ugb 

The upper surface 62a of the upper rail 62 is downwardly * e ^-angle CCD cameras 58 are mounted on the frame 

inclined and the upper surface 62a of the lower rail is » 50 behind the close-up CCD cameras 54 the wide-angle 

upwardly inclined so that the CCD cameras 54 supported by CCD camer ^ 58 have ™***™*«» fields of view. As such 

the rails are angled towards the electronic display device 12. "f "» te ^ultaneousty by both arrays of 

_ „° , „ CCD cameras. Furthermore, it has been found that the 

The outer CCD cameras 54 of the middle row of array 52 mcIined orientation of mc CCD cameras S4 improves res0 . 

are supported by carnages 60 mounted on stubs 70. Each ^non as compared with in-line oriented CCD cameras due 

stub 70 is held by a drive 66 moveable : along a respective rai to me Qature of ^ electronic display 

v 10 ?™ oonttoU %f 8 J? ^ e ft Ver ! ,C ^ The wobulator 18 is mounted on the frame 50 adjacent the 

position of these CCD cameras 54. The stubs 70 extend . t . ,. t , . ~, , . . - c . , , 

f . - ££ .u . *u electromc display device 12. The wobulator 18 includes a 

inwardly and rearwardly from the drives 66 so that the CCD . - 1 4 ,f 7 , „ - , , 

*a - j u . u- i ij. „i pair of laterally spaced, generally vertical coils as shown in 

cameras 54 supported by the stubs are also angled towards £ T „ A * c 1* n j n u ■ * i 

.u i - * X i j • i t*u * 35 FIG. 2a. A pair of vertically spaced, generally horizontal 

the electronic display device 12. The central CCD camera 54 , , .... , . \r r / ? . . . 

- lL „ r *. , « • . coils may also be utilized in the wobulator 18 but are not 

of (he row of amy fli^ed. The coils are secured to the frame 50 and 

mounted on a cantilever 74 by way of a hmge 75. Cantilever { ^ ^ mat m m outside of toe fields of ^ of 

74 extends from a face plate 76 secured to the back of frame me CCD M ^ d 5JJ ^ coUs ^ WQUnd aQd 

40 dimensioned to create symmetrical alternating magnetic 

Each of the carriages 60 accommodates a drive 78 which fiel(fej when energized, thai are sufficient to encompass 

is responsive to motor controller 68 to move the carriage 60 substantially the electronic display device 12 under test and 

along the rail 62 or stub 70 and therefore, adjust the lateral bend elcctron bcams ^thin the electronic display device, 

position of the CCD camera 54 supported by the carriage As ^ m r 20 indudes a number of routines 

can be seen m ETG. 3, each dnve 78 includes a motor 80 for 45 wfaich m executcd ^ ^ of thc test ^ ali 

rotating a shaft 82 extending into a channel 84 in the rail 62 mcnt m 1Q Iq dcular) me 00,^^ 20 includes a 

or stub 70 by way of a transmission 85. A pinion 86 is routinc whkh CCD M itiofl 

mounted on ^theshaft82 and K^toriw^faaw*} mformation for mc various sizcd electronic display devices 

84. The pinion en^ges a rack 88 formed on a wall of the rail whicfa are tQ be tested b the test and alignment system 10. 

62orsmb70.Guides90and92exle n dmgfxommecam^ 5Q wfae|| ^ utef 20 ej£eaUes ^ routme> the 

60 are accommodated by complimentary grooves 94 and 96 com ter hcs m 1 10 the motor coa troller 68 which in 

formed in the rails 62 and stubs 70 to guide the carnages 60 mrn ovi6&s { iQ ^ 6fi ^ ?8 10 move tfae raik 

as they move laterally along the rails and stubs. $2 smbs 70 and/of 60 aQd ^ to ^ CCD 

Unlike carriages 60, carriage 72 is not lateraUy adjustable. cameras 54 at the proper desired positions. The computer 20 

Carriage 72 however supports a drive 100 responsive to 55 also includes conventional Image Analysis Software to 

motor controller 68 to pivot the carriage relative to the determine if the CCD cameras 54 and 58 are in focus. If the 

cantilever 74 by way of hinge 75 and adjust its position and CCD cameras are not in focus, the computer 20 provides 

hence the position of the central CCD camera 54. Drive 100 oulput to me motor controller 68 which in turn operates 

is best shown in FIG. 4 and includes a motor 92 and a motors 120 to adjust the positions of the CCD chips 112 by 

transmission 93 for rotating a lead screw 102 cooperating 60 way G f lead-screw assemblies 114. The computer also 

with a stationary nut 104 mounted on the cantilever 74. includes a test and alignment routine which when executed 

Referring back to FIG. 3, one of the CCD cameras 54 is by the computer 20 allows the video image signal output of 

better illustrated. As can be seen, the CCD camera is the CCD cameras 54 and 58 that is received by the computer 

equipped with a lens 110, such as a Tamron 50 MM lens, to be processed and a series of tests to be performed on the 

sold under model number 402026 by Tamron Corp. of Japan. 65 electronic display device. The series of tests performed by 

A CCD chip 112 is disposed behind the lens 110 at a desired test and alignment system 10 is similar to the series of tests 

distance so that a certain small detail of the entire image disclosed in PCT application serial No. PCT/CA95/00352 



12/17/2003, EAST Version: 1.4.1 



5,9< 

7 

filed on Jun. 13, 1995 designating the United States, the 
contents of which are incorporated herein by reference. 

Calibration of the Wide-Angle Cameras 

The fields of view of the wide-angle color CCD cameras 
58 of array 56 must be calibrated to identify and store for 
reference, the relationship of camera pixel locations to the 
corresponding points in their respective fields of view, and 
further to ensure that their fields of view overlap and 
encompass the desired area of interest. The preferred method 
to achieve the above will now be described. 

Referring now to FIG. 11, a calibration grid 130 is shown 
in negative form for ease of illustration. Grid 130 is in the 
form of a sheet having a plurality of rows of symmetrically 
geometrically shaped, for example rectangular, or in the 
preferred form, circular, apertures 131 spaced equally in 
each axis therethrough. The apertures 131 are formed in the 
sheet with precision, so that the distances between the 
centers of the adjacent apertures are known to a high degree 
of accuracy. A preferred method of creating grid 130 is by 
computer aided design (CAD) methods and further by 
photoplotting the design on high stability photographic film. 
For the calibration process, suitable back-lighting is applied. 
Although the grid 130 is illustrated in negative form, those 
of skill in the art will appreciate that in reality the apertures 
131 are transparent and the surrounding areas of the grid 130 
are opaque. 

At spaced locations on the sheet, corresponding to the 
spacing of the wide-angle CCD cameras 58 in the array 56, 
the apertures 132 are identifiably different from all other 
apertures in the grid 130. For example, these special aper- 
tures may have different shapes or sizes. In the preferred 
embodiment, they are differentiated in effect by size to the 
extreme, by omitting them entirely. This results in larger, 
generally rectangular opaque areas 132 being provided on 
the grid 130. 

During the computer image analysis process of the cali- 
bration grid, the location of the omitted apertures 132 are 
calculated by interpolating the coordinates of the adjacent 
apertures 133. These locations of the omitted apertures 132 
are regarded as the central points of the region of interest 
134 for each of the wide-angle CCD cameras 58. Since the 
distance between adjacent apertures in each row is known, 
the field of view of each CCD camera 58 is determined by 
counting the number of apertures 131 on grid 130 within the 
field of view along the four orthogonal directions extending 
from the centers of the field of view. Along each adjacent 
edge of the fields of view of adjacent CCD cameras 58 is 
located a row 135 or a column 136 of apertures 131 that is 
within the regions of interest of both adjacent CCD cameras. 
Thus their regions of interest 134 overlap. This arrangement 
facilitates the accurate spatial connections of areas viewed 
by the separate wide-angle color CCD cameras 58 of array 
56, so that they encompass the entire desired area of interest. 

During the application of the system 10, the electronic 
display device 12 is located in the generally desired location 
in relation to test fixture 14, where the calibration was 
performed using the grid 130. The test pattern applied to the 
electronic display device 12 for the evaluation or alignment 
of the image geometry displayed thereon is constructed of 
generally rectangularly shaped dots such that some of the 
dots fall within the overlapping areas of adjacent CCD 
cameras 58. By correlating the locations of the dots viewed 
by the adjacent CCD cameras, and further applying trigo- 
nometric calculations that are known to those of skill in the 
art, the exact location of dots displayed by the electronic 
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display device 12 can be identified in all three dimensions. 
Furthermore, by including the geometric constants relating 
to the electronic display device 12, such as the radius of 
curvature and the thickness and refraction constant of glass 
5 utilized, corrections can be made for all major error sources 
by computer calculations. In effect, the CCD cameras 58 of 
array 56 can analyze the displayed image as if it were 
viewed from an infinite distance, which is advantageous for 
the definition of geometric errors. 

10 

Test and Alignment System Operation 

The operation of the test and alignment system will now 
be described. Initially, the computer 20 executes the set-up 
routine to position the CCD cameras 54 at the proper 

15 positions so that the fields of view of the CCD cameras are 
directed towards the electronic display device 12 to be tested 
and aligned. Following that, the computer 20 conditions the 
test pattern generator 26 to feed test patterns to the electronic 
display device. The electronic display device 12 in turn 

20 displays the test patterns. The CCD cameras 54 and 58 
which are maintained in focus, capture images of the dis- 
played test patterns. The close-up CCD cameras 54 focus in 
on small details of the displayed image and therefore, 
capture small areas of the entire test pattern displayed on the 

25 electronic display device 12. On the other hand, the wide- 
angle CCD cameras 58, which have overlapping fields of 
view, collectively capture an image of the entire test pattern 
displayed on the electronic display device. 

3Q The video image signals output by the CCD cameras 54 
and 58 are multiplexed and processed by the multiplexer and 
high speed video input channel processor 22. The high speed 
video input channel processor 22 performs a variety of 
fundamental image processing operations on the video 

35 image signals including digitizing after multiplexing, 
integration, thresholding, interpolation, histogram analysis 
as well as horizontal and vertical projections. The video 
input channel processor also converts the video image 
signals from luminance and chrominance values to red, 

^ green and blue values for processing by the computer 20. 
As mentioned previously, the computer 20 analyses the 
video image signal output of the CCD cameras 54 and 58 to 
perform a comprehensive series of tests on the electronic 
display device when executing the test and alignment rou- 

4 5 tine. The modular nature of the test and alignment routine 
allows an operator to condition the computer 20 to perform 
the tests in virtually any order and re-perform one or more 
of the tests if desired. The various tests which the computer 
20 can perform will now be described with reference to 

50 FIGS. 5 to 10. 

Focus 

To measure focus, the video image signals output by the 
close-up CCD cameras 54 are analyzed by computer 20. A 

55 technique in software is used involving first measuring line 
width for the spot size and then measuring the depth of 
contrast which is a Modulation Transfer Function (MTF). 
Other focus determination methods based on analysis of 
pixel intensity histogram or dimensions of small features 

60 displayed on the electronic display device 12 such as dots, 
may be applied. By determining the center of the mass and 
measuring the spot size at the center of the depth of contrast, 
the dot correlates very well to the human eye's perception of 
good focus. During this test, the computer 20 generates a 

65 graphical panel for display on the operator display 24 
together with a "PASS" or "FAIL" readout, as shown in FIG. 
5. The focussing voltage applied to the electronic display 
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device 12 may be adjusted by the scan unit 32 to improve the rings to bring the red and blue crosses as close to the center 

focus of the electronic display device. The electronic display of the crosshairs as possible. During this test, the central 

device under test is considered to be focused within the CCD camera 54 is positioned at the center of the electronic 

specified limits when the red "ball** is located within the display device and the video image signals generated by the 

dotted rectangle. The closer the red "ball" is to the center of 5 CCD camera 54 are analyzed and processed by the computer 

the crosshairs the better the focus of the electronic display 20. The static convergence of the electronic display device 

device 12. under test, as an example, is acceptable when the red and 

blue crosses are adjusted so that they are located within the 

Color Purity/Vertical Raster Shift/Yoke Rotation dotted rec tangle and no further than half of the width of the 

During the "Color Purity/Vertical Raster Shift/Yoke Rota- 10 dotted rectangle apart, 

tion" test, the computer 20 generates a graphical panel as . 

shown in FIG. 6 for display on the operator display 24. The y 6 

graphical panel includes several video meters which indicate During the Dynamic Convergence test, the computer 20 

adjustments to the yoke and two purity rings of the elec- generates a graphical panel as shown in FIG. 8, for display 

tronic display device. As the operator moves the yoke 15 on the operator display 24. The graphical panel illustrates 

backward and forward, the "balls** in both of the upper, convergence as typically measured at four or more locations 

outside square-boxed meters move from the outside towards on the electronic display device. During this test, the video 

the center of the screen and back and the lower middle meter image signals from a number of the CCD cameras 54 are 

pointers move up and down. Rotation of the yoke will analyzed and processed by the computer 20. These four 

register in both of the lower outside "Tilt" meters. Move- 20 locations in the example are along the horizontal and vertical 

ment of the two purity rings will register in all three of the axes of the electronic display device, near to the edge of the 

upper meters. As well, the upper middle meter will indicate active display area. A combined result is displayed in the 

vertical raster shift related to the top to bottom position of center meter. If the "ball" is within the dotted rectangle, the 

the indicator "ball". The operator display 24 also displays a four convergence measurements are within the specified 

"PASS" or "FAIL" readout dependent on whether or not the 25 limits. Yoke tilt is also displayed. If both Yoke tilt and 

adjustment specifications have been met. measured convergence are within specified limits, a "PASS** 

During color purity and luminance measurements, the result is obtained, 
wobulator 18 is energized to bend electron beams within the During both convergence measurements, the individual 
electronic display device 12 as the images are being cap- dots of color on the electronic display device cannot be 
turcd by the CCD cameras 54 and 58. Specifically, the analyzed by themselves for two reasons. The color dots on 
vertical coils of wobulator 18 are simultaneously energized an electronic display device are not perfectly round but 
by an alternating polarity current. The coils generate a typically have very odd shapes, and due to the large cross- 
vertical symmetrical alternating magnetic field which section of the electron beam, a group of dots are illuminated, 
encompasses the electronic display device and deflects To compensate for irregularity in beam shape, a group of 
horizontally, the electron beams impinging upon the phos- 35 dots for each color is analyzed and the relative position of 
phors in the electronic display device. The alternating sym- the centers of their masses are calculated mathematically. By 
metrical deflection of the electron beams in relation to the analyzing the projection, the irregular beam shapes can be 
phosphor dots or phosphor stripes in the electronic display compensated for to provide a more accurate measurement, 
device result in equal reduction of emitted light intensity 

provided the electron beams are aligned with the center of 40 Orthogonality 

the phosphor dots or stripes when no magnetic field is The system 10 also tests the orthogonality of the hori- 

generated by the wobulator 18. Any difference in light zontal and vertical windings of the deflection yoke and 

intensity is an indication of discrepancy in beam landing. displays a graphical panel on operator display 24 which 

Following this, the horizontal coils of wobulator 18 (if 4S includes an "X" and "Y" gauge as well as a "PASS" or 

included) are simultaneously energized to generate a sym- "FAIL*' readout, as shown in FIG. 9. The method used to 

metrical alternating horizontal magnetic field which encom- determine the orthogonality of the windings is well known 

passes the electronic display device and deflects vertically, to those of skill in the art. 
the electron beams impinging upon the phosphor dots or 

stripes in the electronic display device. so 

While the wobulator coils are being energized, the video Similar to orthogonality, a variety of other geometric tests 

image signals generated by the close-up CCD cameras 54 that are well known to those of skill in the art can be 

are analyzed and processed by the computer 20 to determine performed on the electronic display device 12 by the test and 

beam landing discrepancy. If desired, the video image alignment system 10 such as for example, pincushion and 

signals generated by the wide-angle CCD cameras 58, or a 55 gull-wing distortion, linearity etc. 

selected Region of Interest thereof, can be analyzed and As mentioned previously, if desired, one or more of the 

processed by the computer 20 to determine beam landing above tests can be re-performed at any time during execu- 

discrepancy. tion of the test and alignment routine. For example, FIG. 10 

shows the graphical panel output by computer 20 for display 

Static Convergence 6Q 0Q lhe operator pane i 24 during re-performance of the 

During the Static Convergence test, the computer 20 "Color Purity/Vertical Raster Shift/Yoke Rotation" test, 

generates a graphical panel as shown in FIG. 7, for display As those of skill in the art will appreciate, the test and 

on the operator display 24. The graphical panel reflects the alignment system allows close-up and wide -area images of 

result of adjustments to the magnetic correction device of test patterns displayed on an electronic display device to be 

the electronic display device, traditionally two pairs of yoke 65 captured simultaneously permitting a series of tests to be 

rings known as the four pole ring magnets and the six pole performed on the electronic display device quickly and in 

ring magnets. The operator adjusts the two pairs of yoke real-time. 



Other Geometric Features 
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Although a preferred embodiment of the present invention 13. A test and alignment system for an electronic display 

has been described, those of skill in the art will appreciate device comprising: 

that other variations and modifications may be made without a test pattern generator to be connected to an electronic 

departing from the scope thereof as defined by the appended display device for causing said electronic display 

claims. 5 device to generate images of video test patterns on said 

We claim: electronic display device; 

1. A test fixture for a test and alignment system for an a test fixture including a frame supporting a plurality of 
electronic display device comprising: close-up optical sensors to sense and produce image 

a frame; signals corresponding to small areas of images dis- 

a plurality of close-up optical sensors mounted on said 10 played on said electronic display device and a plurality 

frame to sense and produce image signals correspond- of wide-angle optical sensors to sense and produce 

ing to small areas of an image displayed on an elec- image signals corresponding to larger areas of images 

tronic display device positioned adjacent said test fix- displayed on said electronic display device, said close - 

ture; and up optical sensors being positioned on said frame so as 

a plurality of wide-angle optical sensors mounted on said to be outside of the fields of view of said wide-angle 

frame behind said close-up optical sensors, said wide- optical sensors, at least some of said close-up optical 

angle optical sensors being calibrated using a calibra- sensors being moveable relative to said frame, said 

tion grid so as to have overlapping fields of view to wide-angle optical sensors being calibrated using a 

sense and produce image signals corresponding to calibration grid so as to have overlapping fields of view 

substantially the entire image displayed on said elec- to sense and produce image signals corresponding to 

tronic display device, wherein said close-up optical 20 substantially the entire image displayed on said elec- 

sensors are positioned on said frame in front of said tronic display device; 

wide-angle optical sensors so as to be outside of the a computer controlling said test pattern generator and 

fields of view of said wide-angle optical sensors, at processing said image signals to perform a series of 

least some of said close-up optical sensors being move- tests on said electronic display device; and 

ably mounted on said frame. 25 a display to provide a visual indication of the results of 

2. A test fixture as defined in claim 1 wherein said said series of tests. 

close-up optical sensors include a plurality of rows of optical 14. A test and alignment system as defined in claim 13 

sensors, said optical sensors being angled in a direction wherein said close-up optical sensors include a plurality of 

towards said electronic display device. rows of optical sensors, said optical sensors being angled in 

3. A test fixture as defined in claim 2 further including at 30 a direction towards said electronic display device. 

least one intermediate central row of optical sensors. 15. A test and alignment system as defined in claim 14 

4. A test fixture as defined in claim 3 wherein said central further including at least one intermediate central row of 
row of optical sensors includes a vertically movable central optical sensors, 

optical sensor, said central optical sensor being positioned 16. A test and alignment system as defined in claim 15 

on said frame so that the field of view thereof includes the 35 wherein said central row of optical sensors includes a 

center of said image. vertically movable central optical sensor, said central optical 

5. A test fixture as defined in claim 4 wherein said central sensor being positioned on said frame so that the field of 
optical sensor is pivotally mounted on said frame. view thereof includes the center of said image. 

6. A test fixture as defined in claim 5 wherein said central 17. A test and alignment system as defined in claim 16 
optical sensor is pivotally mounted on a cantilever. 40 wherein said central optical sensor is pivotally mounted on 

7. A test fixture as defined in claim 2 wherein said rows said frame. 

of optical sensors are supported on inclined rails mounted on 18. A test and alignment system as defined in claim 17 

said frame, said rails being vertically adjustable. wherein said central optical sensor is pivotally mounted on 

8. A test fixture as defined in claim 7 wherein the optical a cantilever. 

sensors in said rows are movable laterally along said respec- 45 19. A test and alignment system as defined in claim 14 

tive inclined rails. wherein said rows of optical sensors are supported on 

9. A test fixture as defined in claim 8 wherein the optical inclined rails mounted on said frame, said rails being ver- 
sensors in said rows include drive means to move said tically adjustable. 

optical sensors laterally along said rails. 20. A test and alignment system as defined in claim 19 

10. A test fixture as defined in claim 1 further including at 50 wherein the optical sensors in said rows are movable later- 
least one wobulator mounted on said frame for creating an ally along said respective inclined rails. 

alternating magnetic field to bend electron beams within said 21. A test and alignment system as defined in claim 20 

electronic display device. wherein the optical sensors in said rows include drive means 

11. A test fixture as defined in claim 10 wherein said at to move said optical sensors laterally along said rails, 
least one wobulator includes at least one pair of laterally 55 22. A test and alignment system as defined in claim 13 
spaced, elongate electromagnets mounted on said frame further including at least one wobulator mounted on said 
outside of the fields of view of said close-up and wide-angle frame for creating an alternating magnetic field to bend 
optical sensors and being energizable to generate a generally electron beams within said electronic display device, 
uniform alternating magnetic field, said alternating magnetic 23. A test and alignment system as defined in claim 22 
field encompassing generally the entire display screen of 60 wherein said at least one wobulator includes at least one pair 
said electronic display device. of laterally spaced electromagnets mounted on said frame 

12. A test fixture as defined in claim U wherein said at and being energizable to generate a generally uniform alter- 
least one wobulator includes a pair of vertically spaced, nating magnetic field, said alternating magnetic field encom- 
generally horizontal electromagnets and a pair of laterally passing generally the entire display screen of said electronic 
spaced, generally vertical electromagnets, said pairs of elec- 65 display device. 

tro magnets being energizable to generate alternating mag- 24. A test and alignment system as defined in claim 23 

netic fields. wherein said at least one wobulator includes a pair of 
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vertically spaced, generally horizontal electromagnets and a 
pair of laterally spaced, generally vertical electromagnets, 
said pairs of electromagnets being energizable to generate 
alternating magnetic fields. 

25. A test fixture for a test and alignment system for an 
electronic display device comprising: 

a frame; 

at least one optical sensor mounted on said frame to sense 
and produce image signals corresponding to an image 
displayed on an electronic display device positioned 
adjacent said test fixture; and 

at least one wobulator mounted on said frame and includ- 
ing at least one pair of laterally spaced electromagnets, 
said electromagnets being energizable to generate a 
generally uniform alternating magnetic field generally 
encompassing the entire display screen of said elec- 
tronic display device to bend electron beams within 
said electronic display device, said electromagnets 
being fixed to said frame outside of the field of view of 
said at least one optical sensor. 

26. A test fixture as defined in claim 25 wherein said at 
least one wobulator includes a pair of vertically spaced, 
generally horizontal electromagnets and a pair of laterally 
spaced, generally vertical electromagnets, said pairs of elec- 
tromagnets being energizable to generate alternating mag- 
netic fields. 

27. A test fixture as defined in claim 1 wherein at least one 
of said wide-angle and close-up optical sensors includes a 
CCD device and a fixed lens element, said CCD device 
being movable relative to said lens element to change the 
focus of said at least one optical sensor. 

28. A test fixture as defined in claim 27 wherein said CCD 
device is mounted on a slide assembly responsive to a drive, 
said slide assembly being slidable along an axis parallel to 
an optical axis extending through said at least one optical 
sensor. 

29. A test and alignment system as defined in claim 13 
wherein at least one of said wide-angle and close-up optical 
sensors includes a CCD device and a fixed lens element, said 
CCD device being movable relative to said lens element to 
change the focus of said at least one optical sensor. 

30. A test and alignment system as defined in claim 29 
wherein said CCD device is mounted on a slide assembly 
responsive to a drive, said slide assembly being slidable 
along an axis parallel to an optical axis extending through 
said at least one optical sensor. 
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31. A test fixture for a test and alignment system for an 
electronic display device comprising: 

a frame; 

5 a plurality of close-up cameras mounted on said frame to 
sense and produce image signals corresponding to 
small areas of an image displayed on an electronic 
display device positioned adjacent said test fixture; and 
a plurality of wide-angle cameras mounted on said frame 
10 behind said close-up cameras and being calibrated 
using a calibration grid so as to have overlapping fields 
of view, said wide-angle cameras sensing and produc- 
ing image signals corresponding to substantially the 
entire image displayed on said electronic display 
device, said close-up cameras being positioned on said 
frame in front of said wide-angle cameras outside of the 
fields of view of said wide-angle cameras. 

32. A test fixture as defined in claim 31 wherein said 
close-up cameras are arranged in an array including a 
plurality of vertically spaced rows, each of said rows includ- 
ing a plurality of close-up cameras, said close-up cameras 
being angled in a direction towards said electronic display 
device to compensate for convexities in the shape of said 

25 electronic display device. 

33. A test fixture as defined in claim 32 wherein said rows 
of close-up cameras are supported on inclined rails mounted 
on said frame, said rails being vertically adjustable. 

34. A test fixture as defined in claim 33 wherein said 
30 close-up cameras are movable along said respective inclined 

rails. 

35. A test fixture as defined in claim 31 wherein each of 
said wide-angle and close-up cameras includes a CCD 
device and a fixed lens element, said CCD device being 

35 moveable relative to said lens element to change the focus 
of said camera. 

36. A test fixture as defined in claim 31 wherein said 
calibration grid includes an identifier for each of said 
wide-angle cameras, each identifier defining a central point 
for the field of view of said respective wide-angle camera. 

37. A test fixture as defined in claim 36 wherein said 
calibration grid includes an array of apertures formed in a 
sheet, each said identifier being constituted by a region of 

45 said sheet devoid of apertures. 
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